2020-03-11T22:20:57.720 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-11T22:20:57.720 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.oqWxi3.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848254 2020-03-11T22:20:57.741 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:20:57.794 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848254 branch: wip-zafman-octo-diag description: rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848254' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 20 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.16503 2020-03-11T22:20:57.794 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-11T22:20:57.794 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-11T22:20:57.795 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-11T22:20:57.796 INFO:teuthology.task.internal:Checking packages... 2020-03-11T22:20:57.810 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-11T22:20:57.811 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-11T22:20:57.811 INFO:teuthology.packaging:ref: None 2020-03-11T22:20:57.811 INFO:teuthology.packaging:tag: None 2020-03-11T22:20:57.811 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-11T22:20:57.811 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:20:57.812 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-11T22:20:58.023 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-11T22:20:58.023 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-11T22:20:58.036 INFO:teuthology.task.internal:no buildpackages task found 2020-03-11T22:20:58.036 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-11T22:20:58.058 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-11T22:20:58.058 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:20:58.107 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:21:08.123 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848254', 'machine_type': 'smithi'} 2020-03-11T22:21:08.141 DEBUG:teuthology.lock.ops:locked ubuntu@smithi016.front.sepia.ceph.com 2020-03-11T22:21:08.169 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:21:08.361 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of centos 8.1 2020-03-11T22:21:08.737 INFO:teuthology.orchestra.console:Power off smithi016 2020-03-11T22:21:08.738 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-11T22:21:08.761 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:08.881 INFO:teuthology.orchestra.console:Power off for smithi016 completed 2020-03-11T22:21:08.982 INFO:teuthology.orchestra.console:Power on smithi016 2020-03-11T22:21:08.982 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-11T22:21:09.005 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:13.030 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:13.240 INFO:teuthology.orchestra.console:Power on for smithi016 completed 2020-03-11T22:21:13.341 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish 2020-03-11T22:23:29.452 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:38.523 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:47.578 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:56.631 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:05.691 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:14.746 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:20.750 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:21.077 INFO:teuthology.orchestra.run.smithi016:> hostname 2020-03-11T22:25:21.386 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016.front.sepia.ceph.com 2020-03-11T22:25:21.386 INFO:teuthology.orchestra.run.smithi016:> grep smithi016.front.sepia.ceph.com /etc/hosts 2020-03-11T22:25:21.401 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16 smithi016.front.sepia.ceph.com smithi016 2020-03-11T22:25:21.402 INFO:teuthology.orchestra.run.smithi016:> sudo hostname smithi016 2020-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi016:> sudo sed -i -e 's/smithi016.front.sepia.ceph.com/smithi016/g' /etc/hosts 2020-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi016:> sudo sed -i -e 's/172.21.15.16/172.21.15.16/g' /etc/hosts 2020-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi016:> sudo sed -i -e 's/smithi016.front.sepia.ceph.com/smithi016/g' /etc/hostname 2020-03-11T22:25:21.696 INFO:teuthology.orchestra.run.smithi016:> sudo hostname smithi016 2020-03-11T22:25:21.767 INFO:teuthology.provision.fog.smithi016:Deploy complete! 2020-03-11T22:25:21.769 DEBUG:teuthology.parallel:result is None 2020-03-11T22:25:21.871 INFO:teuthology.lock.ops:Checking smithi016.front.sepia.ceph.com 2020-03-11T22:25:21.872 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-11T22:25:21.890 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: set os type and version on server 2020-03-11T22:25:21.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:25:21.891 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:21.958 INFO:teuthology.orchestra.run.smithi016:> true 2020-03-11T22:25:22.132 INFO:teuthology.orchestra.run.smithi016:> uname -m 2020-03-11T22:25:22.187 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2020-03-11T22:25:22.188 INFO:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2020-03-11T22:25:22.242 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Linux" 2020-03-11T22:25:22.242 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="8 (Core)" 2020-03-11T22:25:22.242 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2020-03-11T22:25:22.242 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:25:22.242 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="8" 2020-03-11T22:25:22.243 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:25:22.243 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:25:22.243 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2020-03-11T22:25:22.243 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:25:22.243 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:25:22.243 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:25:22.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:22.244 INFO:teuthology.orchestra.run.smithi016.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:25:22.244 INFO:teuthology.orchestra.run.smithi016.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:25:22.244 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:25:22.244 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:25:22.244 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:22.245 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2020-03-11T22:25:22.265 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-11T22:25:22.282 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi016.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsjNogAnUusTBbaGzH8dj5zez4h4OJP7DUOAhQqcUvBZu5PtCpTsn15EMmD0aRBjqyy9BgDG7T3tIZih4BzHwmt4cHsXz4NfVJoNNENeh/aXYIQ4Y5vrMno9fwmUTTdChd/vT0nFUpJy9VRJVV3C0kPq9G/xgwQPf0ZGisYw63lZ734+XajIBwOhaxg0jyRFe6AaLQ+3AVtzM67ERNp3b+zvbHEf2vDFR5NVwmxF45XECLRkL6yG+VWPGCk7AYA8uDzicQiWuHpR+RtZqM3w5go+9jwZFLYlRVRVGN5hDgRjtaiPpMWibXFQuEoKUIpY8xKMEjRWIm9q13gbmmeXlNKBeJ7wUJe8UH4iXhWBFDk8SXZc4Ss2hdFud4jnqmTBJEbEP+jvoN8bVcjKsoaCVmzQS7FQRsb0gTFUoiCkvlGocokEIj+lq5nuYXRUatEvlcXuV8+CfO1rsfYUIyHrI+MrK5Hwh1GHPrP5nDEFURU7DCkHGlvb6iE76OLECOFCE= 2020-03-11T22:25:22.282 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:25:22.322 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-11T22:25:22.326 INFO:teuthology.task.internal:Saving configuration 2020-03-11T22:25:22.336 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-11T22:25:22.348 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-11T22:25:22.365 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-11 22:21:08.131445', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi016.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAfoo2MDVpSsM3ghaOTAXbUgVKAX/YO2fA/AdORD2HPerdZKb0qzBBQ6nc4tUXpYcI7xpEOpHlSXJzr3tD6kqZE=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848254'} 2020-03-11T22:25:22.366 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-11T22:25:22.378 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-11T22:25:22.378 INFO:teuthology.run_tasks:Running task console_log... 2020-03-11T22:25:22.427 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:25:22.428 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-11T22:25:22.437 INFO:teuthology.task.internal:Opening connections... 2020-03-11T22:25:22.437 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2020-03-11T22:25:22.438 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:22.510 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-11T22:25:22.521 INFO:teuthology.orchestra.run.smithi016:> uname -m 2020-03-11T22:25:22.688 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2020-03-11T22:25:22.689 INFO:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2020-03-11T22:25:22.743 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Linux" 2020-03-11T22:25:22.743 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="8 (Core)" 2020-03-11T22:25:22.743 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2020-03-11T22:25:22.743 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:25:22.744 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="8" 2020-03-11T22:25:22.744 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:25:22.744 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:25:22.744 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2020-03-11T22:25:22.744 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:25:22.744 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:25:22.744 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:25:22.745 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:22.745 INFO:teuthology.orchestra.run.smithi016.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:25:22.745 INFO:teuthology.orchestra.run.smithi016.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:25:22.745 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:25:22.745 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:25:22.745 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:22.746 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2020-03-11T22:25:22.761 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-11T22:25:22.787 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-11T22:25:22.797 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-11T22:25:22.797 INFO:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2020-03-11T22:25:22.811 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-11T22:25:22.824 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-11T22:25:22.824 INFO:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2020-03-11T22:25:22.872 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-11T22:25:22.900 INFO:teuthology.run_tasks:Running task kernel... 2020-03-11T22:25:22.932 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-11T22:25:22.932 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-11T22:25:22.932 INFO:teuthology.orchestra.run.smithi016:> uname -r 2020-03-11T22:25:22.946 INFO:teuthology.orchestra.run.smithi016.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:25:22.947 INFO:teuthology.task.kernel:Running kernel on smithi016: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:25:22.947 INFO:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2020-03-11T22:25:25.597 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1 day, 6:59:32 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:25:26.012 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-11T22:25:26.012 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-11T22:25:26.175 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:25:26.176 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:25:26.177 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:25:26.443 INFO:teuthology.orchestra.run.smithi016:> echo no | sudo yum reinstall kernel || true 2020-03-11T22:25:26.858 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1 day, 6:59:33 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:25:27.254 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:25:27.254 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:25:27.254 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2020-03-11T22:25:27.254 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:25:27.255 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2020-03-11T22:25:27.255 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:25:27.255 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:25:27.255 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:27.255 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:25:27.255 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:25:27.255 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:27.256 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 3.1 M 2020-03-11T22:25:27.256 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2020-03-11T22:25:27.256 INFO:teuthology.orchestra.run.smithi016.stderr:Operation aborted. 2020-03-11T22:25:27.343 INFO:teuthology.orchestra.run.smithi016.stdout:Is this ok [y/N]: 2020-03-11T22:25:27.344 INFO:teuthology.orchestra.run.smithi016:> sudo yum reinstall -y kernel || true 2020-03-11T22:25:27.761 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 1 day, 6:59:34 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:25:28.127 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:25:28.129 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:25:28.129 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2020-03-11T22:25:28.130 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:25:28.130 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2020-03-11T22:25:28.130 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:25:28.130 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:25:28.130 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:28.131 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:25:28.131 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:25:28.131 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:28.131 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 3.1 M 2020-03-11T22:25:28.131 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2020-03-11T22:25:28.132 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:25:28.246 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): kernel-4.18.0-147.el8.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2020-03-11T22:25:28.254 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2020-03-11T22:25:28.256 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:25:28.256 INFO:teuthology.orchestra.run.smithi016.stdout:Total 24 MB/s | 3.1 MB 00:00 2020-03-11T22:25:28.280 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:25:28.477 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:25:28.478 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:25:28.562 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:25:28.562 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:25:28.719 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:25:28.896 INFO:teuthology.orchestra.run.smithi016.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-11T22:25:29.014 INFO:teuthology.orchestra.run.smithi016.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:25:29.014 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:29.132 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:29.132 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:25:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:25:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-11T22:25:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:25:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-11T22:25:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:29.619 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalled: 2020-03-11T22:25:29.619 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-11T22:25:29.620 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:29.620 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:25:29.718 INFO:teuthology.orchestra.run.smithi016:> rpm -q kernel --last | head -n 1 2020-03-11T22:25:29.768 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:25:28 PM UTC 2020-03-11T22:25:29.770 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:25:29.770 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-11T22:25:29.770 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-11T22:25:29.771 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-11T22:25:29.771 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-11T22:25:29.771 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2020-03-11T22:25:29.771 INFO:teuthology.orchestra.run.smithi016:> rpm -q kernel --last | head -n 1 2020-03-11T22:25:29.814 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:25:28 PM UTC 2020-03-11T22:25:29.816 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:25:29.816 INFO:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2020-03-11T22:25:29.890 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2020-03-11T22:25:29.891 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2020-03-11T22:25:29.891 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.02 2020-03-11T22:25:29.891 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 78.el8 2020-03-11T22:25:29.892 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2020-03-11T22:25:29.892 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Tue 10 Mar 2020 03:09:09 PM UTC 2020-03-11T22:25:29.892 INFO:teuthology.orchestra.run.smithi016.stdout:Group : System Environment/Base 2020-03-11T22:25:29.892 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 9192691 2020-03-11T22:25:29.892 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2020-03-11T22:25:29.893 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-11T22:25:29.893 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-11T22:25:29.893 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-11T22:25:29.893 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-11T22:25:29.894 INFO:teuthology.orchestra.run.smithi016.stdout:Relocations : (not relocatable) 2020-03-11T22:25:29.894 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : CentOS BuildSystem 2020-03-11T22:25:29.894 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2020-03-11T22:25:29.894 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-11T22:25:29.895 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2020-03-11T22:25:29.895 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2020-03-11T22:25:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:29.895 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-11T22:25:29.896 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-11T22:25:29.896 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-11T22:25:29.896 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2020-03-11T22:25:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:25:29.896 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2020-03-11T22:25:29.898 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-11T22:25:29.898 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 4.18.0-147.el8.x86_64 2020-03-11T22:25:29.898 INFO:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-11T22:25:30.857 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2020-03-11T22:25:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:done 2020-03-11T22:25:32.547 INFO:teuthology.orchestra.run.smithi016:> mktemp 2020-03-11T22:25:32.580 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.fQyixRiPkr 2020-03-11T22:25:32.580 INFO:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fQyixRiPkr 2020-03-11T22:25:32.679 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.fQyixRiPkr 2020-03-11T22:25:32.809 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.fQyixRiPkr is 4KB 2020-03-11T22:25:32.859 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.fQyixRiPkr 2020-03-11T22:25:32.874 INFO:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2020-03-11T22:25:32.967 INFO:teuthology.orchestra.run.smithi016.stdout:291b34331a614822a44a2b4be2021771-0-rescue.conf 2020-03-11T22:25:32.967 INFO:teuthology.orchestra.run.smithi016.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-11T22:25:32.967 INFO:teuthology.orchestra.run.smithi016.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64.conf 2020-03-11T22:25:32.969 INFO:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64 2020-03-11T22:25:33.124 INFO:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2020-03-11T22:26:03.154 INFO:teuthology.misc:Re-opening connections... 2020-03-11T22:26:03.154 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2020-03-11T22:26:03.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:03.154 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:21.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2020-03-11T22:26:21.542 DEBUG:teuthology.misc:waited 18.388174057 2020-03-11T22:26:22.543 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2020-03-11T22:26:22.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:22.543 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:22.545 DEBUG:teuthology.orchestra.remote:timed out 2020-03-11T22:27:22.546 DEBUG:teuthology.misc:waited 79.3918130398 2020-03-11T22:27:23.547 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2020-03-11T22:27:23.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:23.547 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:23.868 INFO:teuthology.orchestra.run.smithi016:> true 2020-03-11T22:27:24.172 DEBUG:teuthology.misc:waited 81.0177590847 2020-03-11T22:27:25.173 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-11T22:27:25.173 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-11T22:27:25.173 INFO:teuthology.orchestra.run.smithi016:> uname -r 2020-03-11T22:27:25.190 INFO:teuthology.orchestra.run.smithi016.stdout:4.18.0-147.el8.x86_64 2020-03-11T22:27:25.191 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-11T22:27:25.191 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-11T22:27:26.192 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-11T22:27:26.192 INFO:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-11T22:27:26.295 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2020-03-11T22:27:26.308 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-11T22:27:26.316 INFO:teuthology.task.internal:Creating test directory... 2020-03-11T22:27:26.316 INFO:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-11T22:27:26.334 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-11T22:27:26.430 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-11T22:27:26.442 INFO:teuthology.task.internal:Creating archive directory... 2020-03-11T22:27:26.443 INFO:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-11T22:27:26.469 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-11T22:27:26.479 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-11T22:27:26.479 INFO:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-11T22:27:26.549 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:27:26.569 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:27:26.570 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-11T22:27:26.582 INFO:teuthology.task.internal:Configuring sudo... 2020-03-11T22:27:26.583 INFO:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-11T22:27:26.645 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-11T22:27:26.660 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-11T22:27:26.660 INFO:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-11T22:27:26.701 INFO:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:27:26.779 INFO:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:27:26.848 INFO:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:27:26.886 INFO:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:27:26.938 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-11T22:27:27.023 INFO:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2020-03-11T22:27:27.143 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-11T22:27:27.463 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-11T22:27:27.534 INFO:teuthology.task.internal:Starting timer... 2020-03-11T22:27:27.534 INFO:teuthology.run_tasks:Running task pcp... 2020-03-11T22:27:27.619 INFO:teuthology.run_tasks:Running task selinux... 2020-03-11T22:27:27.636 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-11T22:27:27.636 INFO:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2020-03-11T22:27:27.707 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: [ OK ] 2020-03-11T22:27:27.709 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-11T22:27:27.709 INFO:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2020-03-11T22:27:27.733 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2020-03-11T22:27:27.734 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive'} 2020-03-11T22:27:27.734 INFO:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-11T22:27:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:27:27.805 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-11T22:27:27.805 INFO:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2020-03-11T22:27:27.876 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-11T22:27:27.895 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-11T22:27:27.895 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-11T22:27:28.113 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-11T22:27:28.121 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-11T22:27:28.952 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-11T22:27:31.172 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-11T22:27:34.549 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-11T22:27:34.587 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:34 +0000 (0:00:00.161) 0:00:00.161 ******* 2020-03-11T22:27:37.282 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-03-11T22:27:37.284 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] 2020-03-11T22:27:37.388 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:37 +0000 (0:00:02.800) 0:00:02.962 ******* 2020-03-11T22:27:37.736 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-11T22:27:37.737 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:27:37.775 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:37 +0000 (0:00:00.386) 0:00:03.349 ******* 2020-03-11T22:27:38.359 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-11T22:27:38.361 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-11T22:27:38.399 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:38 +0000 (0:00:00.624) 0:00:03.973 ******* 2020-03-11T22:27:38.424 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-11T22:27:38.426 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:38.460 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:38 +0000 (0:00:00.061) 0:00:04.034 ******* 2020-03-11T22:27:38.835 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-11T22:27:38.837 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:27:38.874 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:38 +0000 (0:00:00.413) 0:00:04.448 ******* 2020-03-11T22:27:39.926 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-11T22:27:39.928 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583965658.89-269947059209693/source", "state": "file", "uid": 0} 2020-03-11T22:27:39.962 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:39 +0000 (0:00:01.088) 0:00:05.537 ******* 2020-03-11T22:27:40.429 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-11T22:27:40.430 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-11T22:27:40.431 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-11T22:27:40.530 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-11T22:27:40.537 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-11T22:27:40.741 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:40 +0000 (0:00:00.778) 0:00:06.315 ******* 2020-03-11T22:27:41.414 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi016.front.sepia.ceph.com] 2020-03-11T22:27:41.497 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-11T22:27:41.499 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.756) 0:00:07.071 ******* 2020-03-11T22:27:42.082 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-11T22:27:42.083 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.265798", "end": "2020-03-11 22:27:42.051355", "rc": 0, "start": "2020-03-11 22:27:41.785557", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:27:42.120 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.622) 0:00:07.694 ******* 2020-03-11T22:27:42.341 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-11T22:27:42.343 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009269", "end": "2020-03-11 22:27:42.316495", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:27:42.307226", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:27:42.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.259) 0:00:07.954 ******* 2020-03-11T22:27:42.409 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-11T22:27:42.410 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-11T22:27:42.448 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.067) 0:00:08.022 ******* 2020-03-11T22:27:42.475 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-11T22:27:42.476 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.512 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.064) 0:00:08.086 ******* 2020-03-11T22:27:42.539 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-11T22:27:42.540 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.575 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.063) 0:00:08.149 ******* 2020-03-11T22:27:42.603 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-11T22:27:42.605 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.653 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.077) 0:00:08.227 ******* 2020-03-11T22:27:42.679 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-11T22:27:42.680 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.717 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.063) 0:00:08.291 ******* 2020-03-11T22:27:42.753 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-11T22:27:42.754 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.795 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:00.078) 0:00:08.369 ******* 2020-03-11T22:27:42.841 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-11T22:27:42.842 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.853 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.854 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.865 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.866 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.877 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.877 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.889 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.890 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.902 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.912 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.923 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.924 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.935 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.946 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.946 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.958 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.969 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.970 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.981 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.992 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:42.992 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.004 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.017 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.018 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.031 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.041 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.055 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.057 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.068 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.068 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.080 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.081 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.082 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.093 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.106 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.107 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.119 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.120 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.131 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.132 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.144 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.155 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.155 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.167 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.169 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.180 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.181 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.193 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.207 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.208 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.209 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.219 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.230 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.231 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.242 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.242 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.254 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.254 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.265 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.266 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.277 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.278 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.289 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.290 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.305 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.310 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.317 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.318 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.330 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.331 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.342 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.354 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.355 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.356 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.366 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.377 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.378 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.389 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.390 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.401 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.402 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.413 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.413 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.425 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.425 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.437 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.438 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.449 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.460 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.460 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.472 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.472 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.485 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.486 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.497 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.497 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.509 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.509 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.521 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.521 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.532 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.533 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.544 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.556 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.556 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.567 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.568 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.579 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.581 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.591 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.647 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.851) 0:00:09.220 ******* 2020-03-11T22:27:43.672 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-11T22:27:43.710 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.063) 0:00:09.284 ******* 2020-03-11T22:27:43.737 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-11T22:27:43.738 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.779 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.069) 0:00:09.353 ******* 2020-03-11T22:27:43.807 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-11T22:27:43.808 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.847 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.067) 0:00:09.421 ******* 2020-03-11T22:27:43.883 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-11T22:27:43.885 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.922 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.074) 0:00:09.496 ******* 2020-03-11T22:27:43.959 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-11T22:27:43.960 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.002 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.079) 0:00:09.576 ******* 2020-03-11T22:27:44.030 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-11T22:27:44.032 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.068 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:44 +0000 (0:00:00.066) 0:00:09.642 ******* 2020-03-11T22:27:44.104 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-11T22:27:44.104 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.105 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.116 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.116 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.129 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.140 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.141 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.143 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.153 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.164 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.165 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.166 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.176 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.177 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.188 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.188 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.199 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.199 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.210 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.211 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.221 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.222 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.233 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.234 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.244 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.255 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.256 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.267 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.267 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.278 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.279 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.289 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.290 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.291 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.301 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.302 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.312 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.323 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.324 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.334 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.335 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:44 +0000 (0:00:00.311) 0:00:09.954 ******* 2020-03-11T22:27:45.213 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-11T22:27:45.215 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-11T22:27:45.258 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.878) 0:00:10.832 ******* 2020-03-11T22:27:45.295 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-11T22:27:45.296 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => { "msg": "Host smithi016.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-11T22:27:45.336 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.077) 0:00:10.910 ******* 2020-03-11T22:27:45.374 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi016.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-11T22:27:45.413 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.077) 0:00:10.987 ******* 2020-03-11T22:27:45.449 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-11T22:27:45.451 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:45.492 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.078) 0:00:11.066 ******* 2020-03-11T22:27:45.807 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-11T22:27:45.808 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.104648", "end": "2020-03-11 22:27:45.783326", "rc": 0, "start": "2020-03-11 22:27:45.678678", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-11T22:27:45.864 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.371) 0:00:11.438 ******* 2020-03-11T22:27:45.892 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-11T22:27:45.894 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:45.953 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:45 +0000 (0:00:00.088) 0:00:11.527 ******* 2020-03-11T22:27:45.992 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-11T22:27:45.994 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.033 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.080) 0:00:11.608 ******* 2020-03-11T22:27:46.061 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-11T22:27:46.063 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.105 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.071) 0:00:11.679 ******* 2020-03-11T22:27:46.131 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-11T22:27:46.132 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.169 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.064) 0:00:11.743 ******* 2020-03-11T22:27:46.195 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-11T22:27:46.196 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.233 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.063) 0:00:11.807 ******* 2020-03-11T22:27:46.258 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-11T22:27:46.260 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.305 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.072) 0:00:11.879 ******* 2020-03-11T22:27:46.341 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-11T22:27:46.343 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:46.381 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.075) 0:00:11.955 ******* 2020-03-11T22:27:46.406 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-11T22:27:46.408 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.444 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.063) 0:00:12.018 ******* 2020-03-11T22:27:46.480 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-11T22:27:46.482 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.517 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.072) 0:00:12.091 ******* 2020-03-11T22:27:46.542 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-11T22:27:46.544 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:46.579 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.062) 0:00:12.153 ******* 2020-03-11T22:27:46.605 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-11T22:27:46.607 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.642 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.063) 0:00:12.217 ******* 2020-03-11T22:27:46.679 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-11T22:27:46.680 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:46.715 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.072) 0:00:12.289 ******* 2020-03-11T22:27:46.741 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-11T22:27:46.742 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.782 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.066) 0:00:12.356 ******* 2020-03-11T22:27:46.808 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-11T22:27:46.810 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.844 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.062) 0:00:12.418 ******* 2020-03-11T22:27:46.870 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-11T22:27:46.872 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.916 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.071) 0:00:12.490 ******* 2020-03-11T22:27:46.953 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-11T22:27:46.955 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:46.992 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:00.075) 0:00:12.566 ******* 2020-03-11T22:27:47.018 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-11T22:27:47.020 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:47.057 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.065) 0:00:12.631 ******* 2020-03-11T22:27:47.084 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:27:47.085 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:47.122 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.064) 0:00:12.696 ******* 2020-03-11T22:27:47.159 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-11T22:27:47.160 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:47.215 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.092) 0:00:12.789 ******* 2020-03-11T22:27:47.252 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-11T22:27:47.254 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:47.294 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.079) 0:00:12.868 ******* 2020-03-11T22:27:47.331 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-11T22:27:47.332 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:47.372 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.078) 0:00:12.947 ******* 2020-03-11T22:27:47.408 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-11T22:27:47.410 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:47.456 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.083) 0:00:13.030 ******* 2020-03-11T22:27:47.483 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:47.485 INFO:teuthology.task.ansible.out:TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:47.531 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.075) 0:00:13.105 ******* 2020-03-11T22:27:47.533 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:47.559 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:47.561 INFO:teuthology.task.ansible.out:TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:27:47.597 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.066) 0:00:13.171 ******* 2020-03-11T22:27:47.599 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:47.625 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-11T22:27:47.676 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.078) 0:00:13.250 ******* 2020-03-11T22:27:47.715 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-11T22:27:47.716 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:47.756 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:47 +0000 (0:00:00.080) 0:00:13.330 ******* 2020-03-11T22:27:48.011 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-11T22:27:48.012 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:27:48.049 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:48 +0000 (0:00:00.293) 0:00:13.623 ******* 2020-03-11T22:27:48.567 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-11T22:27:48.568 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-11T22:27:49.060 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-11T22:27:49.098 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:49 +0000 (0:00:01.048) 0:00:14.672 ******* 2020-03-11T22:27:49.614 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-11T22:27:49.616 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-11T22:27:50.163 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-11T22:27:50.203 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:01.104) 0:00:15.777 ******* 2020-03-11T22:27:50.240 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:27:50.241 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.278 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.075) 0:00:15.852 ******* 2020-03-11T22:27:50.306 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:27:50.307 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.372 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.094) 0:00:15.947 ******* 2020-03-11T22:27:50.399 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-11T22:27:50.400 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.439 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.066) 0:00:16.013 ******* 2020-03-11T22:27:50.465 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-11T22:27:50.466 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.517 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.078) 0:00:16.091 ******* 2020-03-11T22:27:50.545 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:27:50.546 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.584 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.067) 0:00:16.158 ******* 2020-03-11T22:27:50.610 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-11T22:27:50.612 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.654 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.070) 0:00:16.229 ******* 2020-03-11T22:27:50.682 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-11T22:27:50.683 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.721 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.066) 0:00:16.296 ******* 2020-03-11T22:27:50.757 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-11T22:27:50.759 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.824 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.102) 0:00:16.398 ******* 2020-03-11T22:27:50.849 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-11T22:27:50.851 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:50.889 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:00.065) 0:00:16.463 ******* 2020-03-11T22:27:51.522 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-11T22:27:51.523 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.406786", "end": "2020-03-11 22:27:51.500312", "rc": 0, "start": "2020-03-11 22:27:51.093526", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:27:51.570 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:51 +0000 (0:00:00.681) 0:00:17.144 ******* 2020-03-11T22:27:54.882 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-11T22:27:54.883 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-11T22:27:54.930 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:54 +0000 (0:00:03.359) 0:00:20.504 ******* 2020-03-11T22:27:54.955 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-11T22:27:54.957 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:54.993 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:54 +0000 (0:00:00.063) 0:00:20.567 ******* 2020-03-11T22:27:55.019 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-11T22:27:55.020 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:55.055 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:55 +0000 (0:00:00.061) 0:00:20.629 ******* 2020-03-11T22:27:55.081 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-11T22:27:55.087 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:55.121 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:55 +0000 (0:00:00.065) 0:00:20.695 ******* 2020-03-11T22:27:55.157 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-11T22:27:55.158 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:55.196 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:55 +0000 (0:00:00.074) 0:00:20.770 ******* 2020-03-11T22:27:55.722 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-11T22:27:55.722 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-11T22:27:55.767 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:55 +0000 (0:00:00.571) 0:00:21.341 ******* 2020-03-11T22:27:56.736 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-11T22:27:56.738 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-11T22:27:56.779 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:56 +0000 (0:00:01.012) 0:00:22.353 ******* 2020-03-11T22:27:57.315 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-11T22:27:57.315 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-11T22:27:57.813 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-11T22:27:58.310 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-11T22:27:58.352 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:58 +0000 (0:00:01.572) 0:00:23.926 ******* 2020-03-11T22:27:58.564 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-11T22:27:58.565 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:27:58.609 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:58 +0000 (0:00:00.256) 0:00:24.182 ******* 2020-03-11T22:27:59.123 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-11T22:27:59.125 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-11T22:27:59.607 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-11T22:27:59.608 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:00.087 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-11T22:28:00.143 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:01.534) 0:00:25.717 ******* 2020-03-11T22:28:00.180 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-11T22:28:00.182 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.222 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:00.078) 0:00:25.796 ******* 2020-03-11T22:28:00.258 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-11T22:28:00.259 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-11T22:28:00.296 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:00.074) 0:00:25.870 ******* 2020-03-11T22:28:00.520 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-11T22:28:00.522 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-11T22:28:00.574 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:00.278) 0:00:26.149 ******* 2020-03-11T22:28:00.611 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-11T22:28:00.612 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.622 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.634 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.648 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.690 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:00.115) 0:00:26.264 ******* 2020-03-11T22:28:01.673 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-11T22:28:01.675 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-11T22:28:01.718 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:01.028) 0:00:27.292 ******* 2020-03-11T22:28:01.746 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-11T22:28:01.747 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.792 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.073) 0:00:27.366 ******* 2020-03-11T22:28:02.313 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-11T22:28:02.314 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-11T22:28:02.360 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:02 +0000 (0:00:00.567) 0:00:27.934 ******* 2020-03-11T22:28:02.873 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-11T22:28:02.874 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-11T22:28:02.930 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:02 +0000 (0:00:00.570) 0:00:28.504 ******* 2020-03-11T22:28:02.959 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-11T22:28:02.960 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.020 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.089) 0:00:28.594 ******* 2020-03-11T22:28:03.245 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-11T22:28:03.246 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:28:03.283 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.263) 0:00:28.857 ******* 2020-03-11T22:28:03.570 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-11T22:28:03.571 INFO:teuthology.task.ansible.out:fatal: [smithi016.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-11T22:28:03.608 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.324) 0:00:29.182 ******* 2020-03-11T22:28:03.645 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-11T22:28:03.646 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.693 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.085) 0:00:29.268 ******* 2020-03-11T22:28:04.209 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-11T22:28:04.210 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-11T22:28:04.251 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.557) 0:00:29.825 ******* 2020-03-11T22:28:05.066 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-11T22:28:05.068 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:26:55 UTC", "ActiveEnterTimestampMonotonic": "14298373", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target systemd-journald.socket -.mount tmp.mount basic.target nss-lookup.target systemd-tmpfiles-setup.service var-run.mount sysinit.target system.slice network.target time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:26:55 UTC", "AssertTimestampMonotonic": "14297075", "Before": "shutdown.target getty@tty1.service plymouth-quit.service xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight 2020-03-11T22:28:05.070 INFO:teuthology.task.ansible.out:": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:26:55 UTC", "ConditionTimestampMonotonic": "14297075", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2377", "ExecMainStartTimestamp": "Wed 2020-03-11 22:26:55 UTC", "ExecMainStartTimestampMonotonic": "14298343", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2020-03-11 22:26:55 UTC] ; stop_time=[n/a] ; pid=2377 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/li 2020-03-11T22:28:05.071 INFO:teuthology.task.ansible.out:b/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:26:55 UTC", "InactiveExitTimestampMonotonic": "14298373", "InvocationID": "70d8c6d3337940708d7974d7c4305f45", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127461", "LimitNPROCSoft": "127461", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127461", "LimitSIGPENDINGSoft": "127461", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2377", "MemoryAccounting": "yes", "MemoryCurrent": "1634304", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "Mo 2020-03-11T22:28:05.072 INFO:teuthology.task.ansible.out:untAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:26:55 UTC", "StateChangeTimestampMonotonic": "14298373", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "T 2020-03-11T22:28:05.074 INFO:teuthology.task.ansible.out:imeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:26:55 UTC", "WatchdogTimestampMonotonic": "14298371", "WatchdogUSec": "0"}} 2020-03-11T22:28:05.112 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.861) 0:00:30.686 ******* 2020-03-11T22:28:05.337 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-11T22:28:05.338 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001780", "end": "2020-03-11 22:28:05.304473", "rc": 0, "start": "2020-03-11 22:28:05.302693", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-11T22:28:05.376 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.263) 0:00:30.950 ******* 2020-03-11T22:28:06.356 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:06.357 INFO:teuthology.task.ansible.out:TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-11T22:28:06.358 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-11T22:28:06.395 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:01.019) 0:00:31.970 ******* 2020-03-11T22:28:07.367 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-11T22:28:07.369 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-11T22:28:07.405 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:01.009) 0:00:32.979 ******* 2020-03-11T22:28:08.792 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-11T22:28:08.831 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:01.426) 0:00:34.406 ******* 2020-03-11T22:28:25.855 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-11T22:28:25.857 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.815051", "end": "2020-03-11 22:28:25.827267", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:28:09.012216", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-11T22:28:25.892 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:25 +0000 (0:00:17.060) 0:00:51.466 ******* 2020-03-11T22:28:26.404 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-11T22:28:26.405 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-11T22:28:26.452 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:26 +0000 (0:00:00.560) 0:00:52.026 ******* 2020-03-11T22:28:26.682 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-11T22:28:26.684 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.014396", "end": "2020-03-11 22:28:26.658283", "rc": 0, "start": "2020-03-11 22:28:26.643887", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:26.720 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:26 +0000 (0:00:00.267) 0:00:52.294 ******* 2020-03-11T22:28:26.937 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-11T22:28:26.938 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002109", "end": "2020-03-11 22:28:26.916899", "rc": 0, "start": "2020-03-11 22:28:26.914790", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:26.977 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:26 +0000 (0:00:00.257) 0:00:52.551 ******* 2020-03-11T22:28:43.919 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-11T22:28:43.921 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.672426", "end": "2020-03-11 22:28:43.889655", "rc": 0, "start": "2020-03-11 22:28:27.217229", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:43.961 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:43 +0000 (0:00:16.983) 0:01:09.535 ******* 2020-03-11T22:28:44.169 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-11T22:28:44.171 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-11T22:28:44.241 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.280) 0:01:09.815 ******* 2020-03-11T22:28:44.280 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-11T22:28:44.280 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-11T22:28:44.319 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.078) 0:01:09.894 ******* 2020-03-11T22:28:44.356 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-11T22:28:44.357 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:28:44.401 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.080) 0:01:09.975 ******* 2020-03-11T22:28:44.447 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-11T22:28:44.448 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-11T22:28:44.500 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.099) 0:01:10.074 ******* 2020-03-11T22:28:44.547 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-11T22:28:44.548 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:28:44.590 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.090) 0:01:10.164 ******* 2020-03-11T22:28:44.803 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-11T22:28:44.804 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:28:44.840 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.250) 0:01:10.415 ******* 2020-03-11T22:28:45.042 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-11T22:28:45.043 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-11T22:28:45.081 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:45 +0000 (0:00:00.240) 0:01:10.655 ******* 2020-03-11T22:28:45.106 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-11T22:28:45.108 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:45.146 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:45 +0000 (0:00:00.064) 0:01:10.720 ******* 2020-03-11T22:28:45.172 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-11T22:28:45.173 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:45.213 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:45 +0000 (0:00:00.067) 0:01:10.788 ******* 2020-03-11T22:28:45.445 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-11T22:28:45.446 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-11T22:28:45.489 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:45 +0000 (0:00:00.275) 0:01:11.063 ******* 2020-03-11T22:28:46.254 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-11T22:28:46.255 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-11T22:28:46.256 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:46.297 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:46 +0000 (0:00:00.807) 0:01:11.871 ******* 2020-03-11T22:28:46.334 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-11T22:28:46.335 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi016"}, "changed": false} 2020-03-11T22:28:46.376 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:46 +0000 (0:00:00.079) 0:01:11.950 ******* 2020-03-11T22:28:47.424 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-11T22:28:47.425 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi016.front.sepia.ceph.com", "ansible_hostname": "smithi016", "ansible_nodename": "smithi016"}, "changed": true, "name": "smithi016"} 2020-03-11T22:28:47.464 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:47 +0000 (0:00:01.088) 0:01:13.038 ******* 2020-03-11T22:28:47.961 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-11T22:28:47.961 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-11T22:28:47.998 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:47 +0000 (0:00:00.534) 0:01:13.573 ******* 2020-03-11T22:28:48.509 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-11T22:28:48.510 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-11T22:28:48.549 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:48 +0000 (0:00:00.550) 0:01:14.123 ******* 2020-03-11T22:28:49.182 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-11T22:28:49.183 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-11T22:28:49.219 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:49 +0000 (0:00:00.670) 0:01:14.793 ******* 2020-03-11T22:28:49.729 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-11T22:28:49.730 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-11T22:28:49.769 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:49 +0000 (0:00:00.549) 0:01:15.343 ******* 2020-03-11T22:28:49.981 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:28:49.982 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:28:50.017 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:50 +0000 (0:00:00.248) 0:01:15.591 ******* 2020-03-11T22:28:50.227 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:28:50.228 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-11T22:28:50.263 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:50 +0000 (0:00:00.246) 0:01:15.838 ******* 2020-03-11T22:28:50.496 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:28:50.498 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-11T22:28:50.543 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:50 +0000 (0:00:00.274) 0:01:16.112 ******* 2020-03-11T22:28:51.041 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:51.041 INFO:teuthology.task.ansible.out:TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:28:51.043 INFO:teuthology.task.ansible.out: ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-11T22:28:51.092 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:51 +0000 (0:00:00.554) 0:01:16.666 ******* 2020-03-11T22:28:51.093 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:51.594 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:28:51.596 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-11T22:28:51.633 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:51 +0000 (0:00:00.540) 0:01:17.207 ******* 2020-03-11T22:28:53.604 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:28:53.606 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.764119", "end": "2020-03-11 22:28:53.577074", "rc": 0, "start": "2020-03-11 22:28:51.812955", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:53.643 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:53 +0000 (0:00:02.009) 0:01:19.217 ******* 2020-03-11T22:28:54.175 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-11T22:28:54.176 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-11T22:28:54.661 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-11T22:28:55.141 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-11T22:28:55.182 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:55 +0000 (0:00:01.539) 0:01:20.756 ******* 2020-03-11T22:28:55.707 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-11T22:28:55.707 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-11T22:28:55.750 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:55 +0000 (0:00:00.567) 0:01:21.324 ******* 2020-03-11T22:28:56.271 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-11T22:28:56.271 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-11T22:28:56.747 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-11T22:28:57.232 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-11T22:28:57.272 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:57 +0000 (0:00:01.522) 0:01:22.847 ******* 2020-03-11T22:28:57.850 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-11T22:28:57.852 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.352380", "end": "2020-03-11 22:28:57.816233", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-11 22:28:57.463853", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:28:57.890 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:57 +0000 (0:00:00.617) 0:01:23.465 ******* 2020-03-11T22:28:57.916 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-11T22:28:57.917 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:57.955 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:57 +0000 (0:00:00.064) 0:01:23.529 ******* 2020-03-11T22:28:57.992 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-11T22:28:57.995 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:58.022 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:58.059 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:58 +0000 (0:00:00.104) 0:01:23.634 ******* 2020-03-11T22:28:58.097 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-11T22:28:58.099 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:58.145 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:58 +0000 (0:00:00.085) 0:01:23.719 ******* 2020-03-11T22:28:58.181 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:28:58.183 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-11T22:28:58.219 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:58 +0000 (0:00:00.073) 0:01:23.793 ******* 2020-03-11T22:28:58.265 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-11T22:28:58.266 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:58.303 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:58 +0000 (0:00:00.083) 0:01:23.877 ******* 2020-03-11T22:28:58.338 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-11T22:28:58.340 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:58.379 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:58 +0000 (0:00:00.076) 0:01:23.953 ******* 2020-03-11T22:28:58.602 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-11T22:28:58.603 INFO:teuthology.task.ansible.out:fatal: [smithi016.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007039", "end": "2020-03-11 22:28:58.578796", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:28:58.571757", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-11T22:28:58.640 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:58 +0000 (0:00:00.260) 0:01:24.214 ******* 2020-03-11T22:28:58.687 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-11T22:28:58.688 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:58.725 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:58 +0000 (0:00:00.084) 0:01:24.299 ******* 2020-03-11T22:29:01.206 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:29:01.207 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-11T22:29:01.247 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:01 +0000 (0:00:02.521) 0:01:26.821 ******* 2020-03-11T22:29:04.284 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:29:04.284 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-11T22:29:04.321 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:04 +0000 (0:00:03.074) 0:01:29.896 ******* 2020-03-11T22:29:09.512 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-11T22:29:09.513 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:04.962825", "end": "2020-03-11 22:29:09.486683", "rc": 0, "start": "2020-03-11 22:29:04.523858", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\""], "stdout": "CentOS-8 - AppStream 136 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 128 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 47 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 167 kB/s | 5.3 kB 00:00 \nlab-extras 1.0 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:02 ago on Wed 11 Mar 2020 10:29:05 PM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-11T22:29:09.514 INFO:teuthology.task.ansible.out:===================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 136 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 128 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 47 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 167 kB/s | 5.3 kB 00:00 ", "lab-extras 1.0 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:02 ago on Wed 11 Mar 2020 10:29:05 PM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-11T22:29:09.551 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:09 +0000 (0:00:05.229) 0:01:35.125 ******* 2020-03-11T22:29:17.636 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:29:17.637 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-11T22:29:17.673 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:08.122) 0:01:43.247 ******* 2020-03-11T22:29:17.709 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-11T22:29:17.710 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.761 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.088) 0:01:43.335 ******* 2020-03-11T22:29:18.755 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:29:18.757 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-11T22:29:18.797 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:01.035) 0:01:44.371 ******* 2020-03-11T22:29:18.833 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:18.834 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.870 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.073) 0:01:44.444 ******* 2020-03-11T22:29:19.221 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-11T22:29:19.223 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice polkit.service dbus.socket basic.target dbus.service sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service iptables.service shutdown.target ipset.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-11T22:29:19.224 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127461", "LimitNPROCSoft": "127461", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127461", 2020-03-11T22:29:19.226 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127461", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-11T22:29:19.227 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-11T22:29:19.261 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.390) 0:01:44.835 ******* 2020-03-11T22:29:19.590 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-11T22:29:19.591 INFO:teuthology.task.ansible.out:fatal: [smithi016.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-11T22:29:19.646 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:00.384) 0:01:45.220 ******* 2020-03-11T22:29:20.535 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-11T22:29:20.536 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-11T22:29:20.578 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.932) 0:01:46.153 ******* 2020-03-11T22:29:20.615 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-11T22:29:20.617 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.653 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.074) 0:01:46.227 ******* 2020-03-11T22:29:20.681 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-11T22:29:20.723 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.069) 0:01:46.297 ******* 2020-03-11T22:29:20.754 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-11T22:29:20.757 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.803 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.080) 0:01:46.378 ******* 2020-03-11T22:29:20.829 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-11T22:29:20.831 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.870 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.066) 0:01:46.444 ******* 2020-03-11T22:29:20.927 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-11T22:29:20.928 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.938 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.979 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.109) 0:01:46.553 ******* 2020-03-11T22:29:21.005 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-11T22:29:21.006 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.047 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.068) 0:01:46.621 ******* 2020-03-11T22:29:21.074 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-11T22:29:21.120 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.072) 0:01:46.694 ******* 2020-03-11T22:29:21.146 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-11T22:29:21.148 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.186 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.065) 0:01:46.760 ******* 2020-03-11T22:29:21.213 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-11T22:29:21.214 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.251 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.065) 0:01:46.826 ******* 2020-03-11T22:29:21.277 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-11T22:29:21.279 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.317 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.065) 0:01:46.891 ******* 2020-03-11T22:29:21.343 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:21.344 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.062) 0:01:46.954 ******* 2020-03-11T22:29:21.381 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:21.406 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-11T22:29:21.407 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.444 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.064) 0:01:47.018 ******* 2020-03-11T22:29:21.480 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:29:21.482 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.518 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.074) 0:01:47.092 ******* 2020-03-11T22:29:21.544 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-11T22:29:21.546 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.581 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.062) 0:01:47.155 ******* 2020-03-11T22:29:21.608 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-11T22:29:21.609 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.645 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.064) 0:01:47.219 ******* 2020-03-11T22:29:21.671 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-11T22:29:21.672 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.708 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.063) 0:01:47.283 ******* 2020-03-11T22:29:21.735 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-11T22:29:21.736 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.773 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.064) 0:01:47.348 ******* 2020-03-11T22:29:21.810 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-11T22:29:21.811 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.851 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.077) 0:01:47.425 ******* 2020-03-11T22:29:21.888 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-11T22:29:21.889 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.925 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.073) 0:01:47.499 ******* 2020-03-11T22:29:21.951 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-11T22:29:21.952 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:21.995 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.070) 0:01:47.569 ******* 2020-03-11T22:29:22.020 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-11T22:29:22.022 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.058 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.063) 0:01:47.633 ******* 2020-03-11T22:29:22.084 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:29:22.086 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.132 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.073) 0:01:47.706 ******* 2020-03-11T22:29:22.158 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:29:22.159 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.195 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.062) 0:01:47.769 ******* 2020-03-11T22:29:22.232 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:29:22.233 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.268 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.073) 0:01:47.843 ******* 2020-03-11T22:29:22.309 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:29:22.310 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.344 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.075) 0:01:47.918 ******* 2020-03-11T22:29:22.369 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:29:22.371 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.406 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.062) 0:01:47.981 ******* 2020-03-11T22:29:22.432 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:29:22.434 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.470 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.063) 0:01:48.044 ******* 2020-03-11T22:29:22.495 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:29:22.497 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.532 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.062) 0:01:48.106 ******* 2020-03-11T22:29:22.558 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:29:22.559 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.596 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.063) 0:01:48.170 ******* 2020-03-11T22:29:22.621 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:29:22.623 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.658 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.062) 0:01:48.232 ******* 2020-03-11T22:29:22.683 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:29:22.685 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.720 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.061) 0:01:48.294 ******* 2020-03-11T22:29:22.745 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:22.747 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.785 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.064) 0:01:48.359 ******* 2020-03-11T22:29:22.811 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:29:22.812 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:22.813 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:22.849 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:22 +0000 (0:00:00.063) 0:01:48.423 ******* 2020-03-11T22:29:23.083 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:23.084 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-11T22:29:23.121 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.272) 0:01:48.695 ******* 2020-03-11T22:29:23.358 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:23.360 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:29:23.400 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.278) 0:01:48.974 ******* 2020-03-11T22:29:23.618 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:23.619 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-11T22:29:23.678 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.278) 0:01:49.252 ******* 2020-03-11T22:29:23.704 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:29:23.706 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:23.752 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.073) 0:01:49.326 ******* 2020-03-11T22:29:23.778 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:29:23.780 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:23.818 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.065) 0:01:49.392 ******* 2020-03-11T22:29:23.848 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:29:23.849 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:23.904 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.085) 0:01:49.478 ******* 2020-03-11T22:29:23.933 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:23.935 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:23.971 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:23 +0000 (0:00:00.066) 0:01:49.545 ******* 2020-03-11T22:29:23.997 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:23.998 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.043 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.072) 0:01:49.617 ******* 2020-03-11T22:29:24.070 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:24.071 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.115 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.071) 0:01:49.689 ******* 2020-03-11T22:29:24.141 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:24.143 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.179 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.064) 0:01:49.754 ******* 2020-03-11T22:29:24.206 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:24.208 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.255 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.075) 0:01:49.830 ******* 2020-03-11T22:29:24.285 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:24.286 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.327 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.071) 0:01:49.901 ******* 2020-03-11T22:29:24.353 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-11T22:29:24.354 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.391 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.064) 0:01:49.965 ******* 2020-03-11T22:29:24.421 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-11T22:29:24.421 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.470 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.078) 0:01:50.044 ******* 2020-03-11T22:29:24.507 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:29:24.508 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.558 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.088) 0:01:50.132 ******* 2020-03-11T22:29:24.596 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:29:24.602 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.645 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.086) 0:01:50.219 ******* 2020-03-11T22:29:24.672 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-11T22:29:24.673 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.720 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.075) 0:01:50.295 ******* 2020-03-11T22:29:24.747 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-11T22:29:24.749 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.802 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.080) 0:01:50.376 ******* 2020-03-11T22:29:24.829 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.872 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.070) 0:01:50.446 ******* 2020-03-11T22:29:24.898 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-11T22:29:24.900 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:24.940 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:00.068) 0:01:50.514 ******* 2020-03-11T22:29:24.978 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-11T22:29:24.979 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.024 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.084) 0:01:50.598 ******* 2020-03-11T22:29:25.061 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-11T22:29:25.063 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.113 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.088) 0:01:50.687 ******* 2020-03-11T22:29:25.142 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-11T22:29:25.143 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.184 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.071) 0:01:50.758 ******* 2020-03-11T22:29:25.212 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-11T22:29:25.214 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.272 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.087) 0:01:50.846 ******* 2020-03-11T22:29:25.307 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.347 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.075) 0:01:50.922 ******* 2020-03-11T22:29:25.386 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:25.388 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-11T22:29:25.429 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.081) 0:01:51.003 ******* 2020-03-11T22:29:25.478 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-11T22:29:25.479 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.489 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-11T22:29:25.500 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.512 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.525 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.526 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.538 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.593 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.164) 0:01:51.167 ******* 2020-03-11T22:29:25.832 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi016.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.012697", "end": "2020-03-11 22:29:25.797640", "rc": 0, "start": "2020-03-11 22:29:25.784943", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-11T22:29:25.879 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.285) 0:01:51.453 ******* 2020-03-11T22:29:25.916 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-11T22:29:25.917 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.963 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.083) 0:01:51.537 ******* 2020-03-11T22:29:25.999 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-11T22:29:26.000 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:26.038 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:26 +0000 (0:00:00.075) 0:01:51.612 ******* 2020-03-11T22:29:26.065 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-11T22:29:26.066 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:26.102 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:26 +0000 (0:00:00.064) 0:01:51.676 ******* 2020-03-11T22:29:26.128 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-11T22:29:26.130 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:26.165 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:26 +0000 (0:00:00.063) 0:01:51.739 ******* 2020-03-11T22:29:27.139 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-11T22:29:27.141 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-11T22:29:27.182 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:27 +0000 (0:00:01.016) 0:01:52.756 ******* 2020-03-11T22:29:27.229 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:29:27.231 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'block_used': 12964195, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130913, u'size_available': 930328219648, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130913, "block_size": 4096, "block_total": 240095108, "block_used": 12964195, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328219648, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-11T22:29:27.270 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:27 +0000 (0:00:00.087) 0:01:52.844 ******* 2020-03-11T22:29:27.493 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-11T22:29:27.493 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005319", "end": "2020-03-11 22:29:27.462075", "rc": 0, "start": "2020-03-11 22:29:27.456756", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:27.548 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:27 +0000 (0:00:00.277) 0:01:53.122 ******* 2020-03-11T22:29:27.590 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-11T22:29:27.591 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'block_used': 12964195, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130913, u'size_available': 930328219648, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130913, "block_size": 4096, "block_total": 240095108, "block_used": 12964195, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328219648, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:27.647 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:27 +0000 (0:00:00.098) 0:01:53.221 ******* 2020-03-11T22:29:27.673 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-11T22:29:27.713 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:27 +0000 (0:00:00.066) 0:01:53.287 ******* 2020-03-11T22:29:27.904 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-11T22:29:27.905 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HKZ5', u'wwn-0x5000c50087198ecb'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HKZ5', u'wwn': u'0x5000c50087198ecb', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HKZ5-part1', u'wwn-0x5000c50087198ecb-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HKZ5", "wwn-0x5000c50087198ecb"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HKZ5-part1", "wwn-0x5000c50087198ecb-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HKZ5", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:29:27.907 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50087198ecb"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:29.451 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-3TeB9U-vcyd-sAII-T7Ni-MbOL-grUW-AtVZiB', u'nvme-INTEL_SSDPED1D480GA_PHMB7513002C480DGN', u'nvme-nvme.8086-50484d42373531333030324334383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.328784", "end": "2020-03-11 22:29:29.413310", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-3TeB9U-vcyd-sAII-T7Ni-MbOL-grUW-AtVZiB", "nvme-INTEL_SSDPED1D480GA_PHMB7513002C480DGN", "nvme-nvme.8086-50484d42373531333030324334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:28.084526", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGP 2020-03-11T22:29:29.453 INFO:teuthology.task.ansible.out:T data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:30.717 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4ORP8hPoeJD0rfnk1Pjh9ycGXEqYyC239'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.040831", "end": "2020-03-11 22:29:30.681565", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4ORP8hPoeJD0rfnk1Pjh9ycGXEqYyC239"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:29.640734", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:31.991 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4LNbLQnbkfgYECs4fj1VeHaKN5XNzIwG5'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.068806", "end": "2020-03-11 22:29:31.956802", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4LNbLQnbkfgYECs4fj1VeHaKN5XNzIwG5"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:30.887996", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:33.253 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR403NNWcjsvpsk1TV3040oAo2j3VI2YQ7Z'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.050641", "end": "2020-03-11 22:29:33.215881", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR403NNWcjsvpsk1TV3040oAo2j3VI2YQ7Z"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:32.165240", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:34.557 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR46nDeWnF3F3SqYgNWGjumIoCcNwOq7HLx'], u'uuids': [u'd8f18148-7972-4b44-9c60-09c31ca159c0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.057853", "end": "2020-03-11 22:29:34.516862", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR46nDeWnF3F3SqYgNWGjumIoCcNwOq7HLx"], "labels": [], "masters": [], "uuids": ["d8f18148-7972-4b44-9c60-09c31ca159c0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:33.459009", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:35.831 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4ZdhSNK23XFYDv1DvgEPNUkaJungDP12o'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.054025", "end": "2020-03-11 22:29:35.792125", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4ZdhSNK23XFYDv1DvgEPNUkaJungDP12o"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:34.738100", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:35.899 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:08.185) 0:02:01.473 ******* 2020-03-11T22:29:36.092 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-11T22:29:36.093 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HKZ5', u'wwn-0x5000c50087198ecb'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HKZ5', u'wwn': u'0x5000c50087198ecb', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HKZ5-part1', u'wwn-0x5000c50087198ecb-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HKZ5", "wwn-0x5000c50087198ecb"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HKZ5-part1", "wwn-0x5000c50087198ecb-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HKZ5", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:29:36.094 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50087198ecb"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.334 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-3TeB9U-vcyd-sAII-T7Ni-MbOL-grUW-AtVZiB', u'nvme-INTEL_SSDPED1D480GA_PHMB7513002C480DGN', u'nvme-nvme.8086-50484d42373531333030324334383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036605", "end": "2020-03-11 22:29:36.305770", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-3TeB9U-vcyd-sAII-T7Ni-MbOL-grUW-AtVZiB", "nvme-INTEL_SSDPED1D480GA_PHMB7513002C480DGN", "nvme-nvme.8086-50484d42373531333030324334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:36.269165", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 2020-03-11T22:29:36.336 INFO:teuthology.task.ansible.out: 0.033798 s, 3.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.033798 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:36.648 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4ORP8hPoeJD0rfnk1Pjh9ycGXEqYyC239'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.099899", "end": "2020-03-11 22:29:36.615750", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4ORP8hPoeJD0rfnk1Pjh9ycGXEqYyC239"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:36.515851", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0971883 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0971883 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:36.962 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4LNbLQnbkfgYECs4fj1VeHaKN5XNzIwG5'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.094509", "end": "2020-03-11 22:29:36.927864", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4LNbLQnbkfgYECs4fj1VeHaKN5XNzIwG5"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:36.833355", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0918893 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0918893 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:37.275 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR403NNWcjsvpsk1TV3040oAo2j3VI2YQ7Z'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.094047", "end": "2020-03-11 22:29:37.251471", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR403NNWcjsvpsk1TV3040oAo2j3VI2YQ7Z"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:37.157424", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0913912 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0913912 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:37.577 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR46nDeWnF3F3SqYgNWGjumIoCcNwOq7HLx'], u'uuids': [u'd8f18148-7972-4b44-9c60-09c31ca159c0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.095863", "end": "2020-03-11 22:29:37.552150", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR46nDeWnF3F3SqYgNWGjumIoCcNwOq7HLx"], "labels": [], "masters": [], "uuids": ["d8f18148-7972-4b44-9c60-09c31ca159c0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:37.456287", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0931646 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0931646 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:37.972 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4ZdhSNK23XFYDv1DvgEPNUkaJungDP12o'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.095154", "end": "2020-03-11 22:29:37.841370", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-x08qjwvc9SJ1k8maJiVpA8GYgmbUxdR4ZdhSNK23XFYDv1DvgEPNUkaJungDP12o"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:37.746216", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.092442 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.092442 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:38.026 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:02.127) 0:02:03.601 ******* 2020-03-11T22:29:38.332 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-11T22:29:38.334 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.069998", "end": "2020-03-11 22:29:38.303160", "rc": 0, "start": "2020-03-11 22:29:38.233162", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:38.375 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.348) 0:02:03.949 ******* 2020-03-11T22:29:38.631 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-11T22:29:38.632 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.047299", "end": "2020-03-11 22:29:38.609903", "rc": 0, "start": "2020-03-11 22:29:38.562604", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:38.698 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.322) 0:02:04.272 ******* 2020-03-11T22:29:38.729 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-11T22:29:38.793 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.094) 0:02:04.367 ******* 2020-03-11T22:29:38.823 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-11T22:29:38.825 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.869 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.076) 0:02:04.443 ******* 2020-03-11T22:29:38.896 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-11T22:29:38.898 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.941 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.071) 0:02:04.515 ******* 2020-03-11T22:29:38.967 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:29:38.968 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.015 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.073) 0:02:04.589 ******* 2020-03-11T22:29:39.052 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-11T22:29:39.052 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.102 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.087) 0:02:04.676 ******* 2020-03-11T22:29:39.315 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-11T22:29:39.317 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:39.365 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.263) 0:02:04.940 ******* 2020-03-11T22:29:39.403 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:29:39.403 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'block_used': 12964195, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130913, u'size_available': 930328219648, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130913, "block_size": 4096, "block_total": 240095108, "block_used": 12964195, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328219648, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.445 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.080) 0:02:05.020 ******* 2020-03-11T22:29:39.481 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-11T22:29:39.482 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.522 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.076) 0:02:05.096 ******* 2020-03-11T22:29:39.558 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:39.560 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.603 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.081) 0:02:05.178 ******* 2020-03-11T22:29:39.630 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:39.631 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.679 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.075) 0:02:05.253 ******* 2020-03-11T22:29:39.716 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-11T22:29:39.717 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.775 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.096) 0:02:05.349 ******* 2020-03-11T22:29:39.803 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.843 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.067) 0:02:05.417 ******* 2020-03-11T22:29:39.869 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-11T22:29:39.871 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.906 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.062) 0:02:05.480 ******* 2020-03-11T22:29:40.550 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-11T22:29:40.551 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-11T22:29:40.591 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.685) 0:02:06.165 ******* 2020-03-11T22:29:41.166 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-11T22:29:41.168 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:41.510 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:41.872 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:42.223 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:42.556 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:42.605 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:02.013) 0:02:08.179 ******* 2020-03-11T22:29:42.817 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:29:42.818 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-11T22:29:42.855 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.250) 0:02:08.430 ******* 2020-03-11T22:29:42.902 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-11T22:29:42.904 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:43.112 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:43.321 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:43.519 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:43.718 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:43.772 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:43 +0000 (0:00:00.916) 0:02:09.346 ******* 2020-03-11T22:29:43.986 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-11T22:29:43.987 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:29:44.042 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:44 +0000 (0:00:00.269) 0:02:09.616 ******* 2020-03-11T22:29:44.078 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-11T22:29:44.079 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-11T22:29:44.125 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:44 +0000 (0:00:00.083) 0:02:09.699 ******* 2020-03-11T22:29:44.502 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-11T22:29:44.504 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true} 2020-03-11T22:29:44.544 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:44 +0000 (0:00:00.418) 0:02:10.118 ******* 2020-03-11T22:29:45.017 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-11T22:29:45.019 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-11T22:29:45.058 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.513) 0:02:10.632 ******* 2020-03-11T22:29:45.094 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-11T22:29:45.095 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.133 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.075) 0:02:10.708 ******* 2020-03-11T22:29:45.159 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-11T22:29:45.160 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.198 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.064) 0:02:10.772 ******* 2020-03-11T22:29:45.223 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-11T22:29:45.225 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.260 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.062) 0:02:10.835 ******* 2020-03-11T22:29:45.297 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-11T22:29:45.298 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.335 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.074) 0:02:10.909 ******* 2020-03-11T22:29:45.361 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-11T22:29:45.362 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.402 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.067) 0:02:10.976 ******* 2020-03-11T22:29:45.427 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-11T22:29:45.429 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.465 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.062) 0:02:11.039 ******* 2020-03-11T22:29:45.490 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-11T22:29:45.492 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.527 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.062) 0:02:11.101 ******* 2020-03-11T22:29:45.552 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-11T22:29:45.554 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.590 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.062) 0:02:11.164 ******* 2020-03-11T22:29:45.615 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-11T22:29:45.617 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.651 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.061) 0:02:11.225 ******* 2020-03-11T22:29:45.687 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-11T22:29:45.688 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.726 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.074) 0:02:11.300 ******* 2020-03-11T22:29:45.762 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-11T22:29:45.763 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:45.802 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:45 +0000 (0:00:00.076) 0:02:11.376 ******* 2020-03-11T22:29:46.311 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-11T22:29:46.313 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-11T22:29:46.366 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.564) 0:02:11.940 ******* 2020-03-11T22:29:46.698 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-11T22:29:46.700 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:26:55 UTC", "ActiveEnterTimestampMonotonic": "14121157", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpd.service tmp.mount basic.target sysinit.target system.slice systemd-tmpfiles-setup.service ntpdate.service -.mount systemd-journald.socket sntp.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:26:54 UTC", "AssertTimestampMonotonic": "13004551", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:26:54 UTC", "ConditionTimestampMonotonic": "13004526", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-11T22:29:46.702 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2325", "ExecMainStartTimestamp": "Wed 2020-03-11 22:26:55 UTC", "ExecMainStartTimestampMonotonic": "13961006", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:26:54 UTC", "InactiveExitTimestampMonotonic": "13008556", "InvocationID": "f1bd215efed44d8eb4d84cb3b8b108e1", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-11T22:29:46.703 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127461", "LimitNPROCSoft": "127461", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127461", "LimitSIGPENDINGSoft": "127461", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2325", "MemoryAccounting": "yes", "MemoryCurrent": "2809856", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-11T22:29:46.704 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:26:55 UTC", "StateChangeTimestampMonotonic": "14121157", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:26:55 UTC", "WatchdogTimestampMonotonic": "13961021", "WatchdogUSec": "0"}} 2020-03-11T22:29:46.739 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.373) 0:02:12.313 ******* 2020-03-11T22:29:46.765 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-11T22:29:46.767 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:46.805 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.063) 0:02:12.377 ******* 2020-03-11T22:29:46.829 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-11T22:29:46.830 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:46.867 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.063) 0:02:12.441 ******* 2020-03-11T22:29:46.894 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-11T22:29:46.896 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:46.931 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:46 +0000 (0:00:00.064) 0:02:12.505 ******* 2020-03-11T22:29:47.146 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-11T22:29:47.146 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-11T22:29:47.208 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.276) 0:02:12.782 ******* 2020-03-11T22:29:47.720 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-11T22:29:47.721 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-11T22:29:47.781 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:47 +0000 (0:00:00.572) 0:02:13.355 ******* 2020-03-11T22:29:49.024 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-11T22:29:49.025 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-11T22:29:49.072 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:01.291) 0:02:14.647 ******* 2020-03-11T22:29:49.098 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-11T22:29:49.100 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:49.134 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.062) 0:02:14.709 ******* 2020-03-11T22:29:49.171 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-11T22:29:49.172 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-03-11T22:29:49.211 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.076) 0:02:14.785 ******* 2020-03-11T22:29:49.422 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-11T22:29:49.423 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-11T22:29:49.458 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.246) 0:02:15.032 ******* 2020-03-11T22:29:49.486 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-11T22:29:49.487 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:49.548 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.090) 0:02:15.123 ******* 2020-03-11T22:29:49.580 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-11T22:29:49.581 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-11T22:29:49.619 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.070) 0:02:15.193 ******* 2020-03-11T22:29:49.652 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:49.692 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.073) 0:02:15.266 ******* 2020-03-11T22:29:49.905 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-11T22:29:49.906 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-11T22:29:49.943 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:49 +0000 (0:00:00.251) 0:02:15.517 ******* 2020-03-11T22:29:50.158 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-11T22:29:50.160 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-11T22:29:50.206 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.262) 0:02:15.780 ******* 2020-03-11T22:29:50.233 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-11T22:29:50.235 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.274 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.067) 0:02:15.848 ******* 2020-03-11T22:29:50.310 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-11T22:29:50.311 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.323 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.324 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.325 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.327 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.378 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.103) 0:02:15.952 ******* 2020-03-11T22:29:50.420 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-11T22:29:50.420 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.422 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.423 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.433 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.434 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.472 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.094) 0:02:16.046 ******* 2020-03-11T22:29:50.504 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.557 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.084) 0:02:16.131 ******* 2020-03-11T22:29:50.583 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-11T22:29:50.585 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.630 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.073) 0:02:16.204 ******* 2020-03-11T22:29:50.657 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-11T22:29:50.658 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.697 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.066) 0:02:16.271 ******* 2020-03-11T22:29:50.723 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-11T22:29:50.724 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.778 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.081) 0:02:16.352 ******* 2020-03-11T22:29:50.815 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-11T22:29:50.816 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.856 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.077) 0:02:16.430 ******* 2020-03-11T22:29:50.882 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:29:50.883 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.924 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.068) 0:02:16.499 ******* 2020-03-11T22:29:50.950 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-11T22:29:50.952 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.990 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:00.065) 0:02:16.565 ******* 2020-03-11T22:29:51.017 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:29:51.018 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:51.055 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:51 +0000 (0:00:00.064) 0:02:16.629 ******* 2020-03-11T22:29:51.299 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-11T22:29:51.300 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005727", "end": "2020-03-11 22:29:51.254732", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-11 22:29:51.249005", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:51.507 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003156", "end": "2020-03-11 22:29:51.463466", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-11 22:29:51.460310", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:51.698 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002814", "end": "2020-03-11 22:29:51.664997", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:29:51.662183", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:52.950 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:01.042413", "end": "2020-03-11 22:29:52.905378", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-11 22:29:51.862965", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:52.996 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:52 +0000 (0:00:01.941) 0:02:18.570 ******* 2020-03-11T22:29:53.023 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-11T22:29:53.070 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.073) 0:02:18.644 ******* 2020-03-11T22:29:53.307 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-11T22:29:53.308 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:29:53.416 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** 2020-03-11T22:29:53.418 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:29:53.419 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-03-11T22:29:53.421 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-03-11T22:29:53.421 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi016.front.sepia.ceph.com : ok=111 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 2020-03-11T22:29:53.421 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.351) 0:02:18.995 ******* =============================================================================== 2020-03-11T22:29:53.422 INFO:teuthology.task.ansible.out:common : nrpe - Remove SELinux policy package -------------------------- 17.06s common : nrpe - Load SELinux policy package ---------------------------- 16.98s testnode : Zap all non-root disks --------------------------------------- 8.19s testnode : Install packages --------------------------------------------- 8.12s testnode : Base group workaround for BZ1782899 -------------------------- 5.23s common : Install Kerberos Packages (RedHat) ----------------------------- 3.36s testnode : Ensure ceph dependency packages are not present. ------------- 3.07s Gathering Facts --------------------------------------------------------- 2.80s testnode : Ensure ceph packages are not present. ------------------------ 2.52s testnode : Blow away lingering OSD data and FSIDs ----------------------- 2.13s testnode : Create logical volume(s) ------------------------------------- 2.01s testnode : Rebuild rpmdb ------------------------------------------------ 2.01s testnode : List any leftover Ceph artifacts from previous jobs ---------- 1.94s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.57s testnode : Configure local mirrorlists ---------------------------------- 1.54s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.53s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.52s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.43s testnode : Install packages via pip ------------------------------------- 1.29s common : Configure local epel mirrorlists ------------------------------- 1.10s 2020-03-11T22:29:53.699 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com')] 2020-03-11T22:29:53.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:29:53.702 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:53.772 INFO:teuthology.orchestra.run.smithi016:> true 2020-03-11T22:29:53.858 INFO:teuthology.run_tasks:Running task clock... 2020-03-11T22:29:53.871 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-11T22:29:53.872 INFO:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-11T22:29:53.938 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-11T22:29:53.959 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-11T22:29:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2020-03-11T22:29:54.026 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2020-03-11T22:29:54.047 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-11T22:29:54.068 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-11T22:29:54.136 INFO:teuthology.orchestra.run.smithi016.stderr:bash: ntpq: command not found 2020-03-11T22:29:54.139 INFO:teuthology.orchestra.run.smithi016.stdout:210 Number of sources = 4 2020-03-11T22:29:54.139 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-11T22:29:54.139 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2020-03-11T22:29:54.139 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:29:54.139 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:29:54.140 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:29:54.140 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:29:54.141 INFO:teuthology.run_tasks:Running task install... 2020-03-11T22:29:54.152 DEBUG:teuthology.task.install:project ceph 2020-03-11T22:29:54.152 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-11T22:29:54.152 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-11T22:29:54.152 INFO:teuthology.task.install:Using flavor: basic 2020-03-11T22:29:54.159 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-11T22:29:54.160 INFO:teuthology.task.install:extra packages: [] 2020-03-11T22:29:54.160 INFO:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2020-03-11T22:29:54.221 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS 2020-03-11T22:29:54.222 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:29:54.223 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-11T22:29:54.223 INFO:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2020-03-11T22:29:54.302 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS 2020-03-11T22:29:54.303 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:29:54.303 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:29:54.550 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-11T22:29:54.551 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-11T22:29:54.772 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-11T22:29:54.773 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-11T22:29:54.818 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-11T22:29:54.818 INFO:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-11T22:29:54.892 INFO:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-11T22:29:54.984 INFO:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-11T22:29:55.015 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2020-03-11T22:29:55.016 INFO:teuthology.orchestra.run.smithi016:> sudo yum clean all 2020-03-11T22:29:55.418 INFO:teuthology.orchestra.run.smithi016.stdout:239 files removed 2020-03-11T22:29:55.448 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw 2020-03-11T22:29:56.189 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 183 kB/s | 69 kB 00:00 2020-03-11T22:29:56.522 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 44 kB/s | 14 kB 00:00 2020-03-11T22:29:56.849 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2020-03-11T22:29:57.008 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-8 - AppStream 47 MB/s | 6.5 MB 00:00 2020-03-11T22:29:58.218 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-8 - Base 42 MB/s | 5.0 MB 00:00 2020-03-11T22:29:59.052 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-8 - Extras 33 kB/s | 2.1 kB 00:00 2020-03-11T22:29:59.205 INFO:teuthology.orchestra.run.smithi016.stdout:Copr repo for ceph-el8 owned by ktdreyer 361 kB/s | 38 kB 00:00 2020-03-11T22:29:59.653 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 15 MB/s | 6.0 MB 00:00 2020-03-11T22:30:00.924 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 5.7 MB/s | 30 kB 00:00 2020-03-11T22:30:01.967 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:30:01.970 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2020-03-11T22:30:01.970 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:01.970 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:30:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-11T22:30:01.971 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2020-03-11T22:30:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-11T22:30:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-11T22:30:01.971 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2020-03-11T22:30:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-11T22:30:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-11T22:30:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-11T22:30:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-11T22:30:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-11T22:30:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-11T22:30:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-11T22:30:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-11T22:30:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-11T22:30:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-11T22:30:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-11T22:30:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-11T22:30:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-11T22:30:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-11T22:30:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-11T22:30:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-11T22:30:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-11T22:30:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-11T22:30:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-11T22:30:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-11T22:30:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:01.975 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:30:01.975 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:01.975 INFO:teuthology.orchestra.run.smithi016.stdout:Install 21 Packages 2020-03-11T22:30:01.975 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2020-03-11T22:30:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:01.976 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 51 M 2020-03-11T22:30:01.976 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:30:02.857 INFO:teuthology.orchestra.run.smithi016.stdout:(1/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 6.2 MB/s | 5.5 MB 00:00 2020-03-11T22:30:02.906 INFO:teuthology.orchestra.run.smithi016.stdout:(2/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 508 kB/s | 25 kB 00:00 2020-03-11T22:30:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:(3/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 7.2 MB/s | 507 kB 00:00 2020-03-11T22:30:03.042 INFO:teuthology.orchestra.run.smithi016.stdout:(4/23): libradosstriper1-15.1.0-1941.g4acc112.e 5.5 MB/s | 377 kB 00:00 2020-03-11T22:30:03.074 INFO:teuthology.orchestra.run.smithi016.stdout:(5/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 8.9 MB/s | 9.8 MB 00:01 2020-03-11T22:30:03.117 INFO:teuthology.orchestra.run.smithi016.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 920 kB/s | 40 kB 00:00 2020-03-11T22:30:03.166 INFO:teuthology.orchestra.run.smithi016.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 998 kB/s | 49 kB 00:00 2020-03-11T22:30:03.221 INFO:teuthology.orchestra.run.smithi016.stdout:(8/23): python3-cephfs-15.1.0-1941.g4acc112.el8 2.9 MB/s | 162 kB 00:00 2020-03-11T22:30:03.286 INFO:teuthology.orchestra.run.smithi016.stdout:(9/23): python3-rados-15.1.0-1941.g4acc112.el8. 5.3 MB/s | 352 kB 00:00 2020-03-11T22:30:03.353 INFO:teuthology.orchestra.run.smithi016.stdout:(10/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 4.5 MB/s | 309 kB 00:00 2020-03-11T22:30:03.418 INFO:teuthology.orchestra.run.smithi016.stdout:(11/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 1.8 MB/s | 113 kB 00:00 2020-03-11T22:30:03.450 INFO:teuthology.orchestra.run.smithi016.stdout:(12/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2020-03-11T22:30:03.455 INFO:teuthology.orchestra.run.smithi016.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 61 MB/s | 259 kB 00:00 2020-03-11T22:30:03.456 INFO:teuthology.orchestra.run.smithi016.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 30 MB/s | 44 kB 00:00 2020-03-11T22:30:03.458 INFO:teuthology.orchestra.run.smithi016.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 36 MB/s | 47 kB 00:00 2020-03-11T22:30:03.584 INFO:teuthology.orchestra.run.smithi016.stdout:(16/23): librgw2-15.1.0-1941.g4acc112.el8.x86_6 12 MB/s | 6.2 MB 00:00 2020-03-11T22:30:03.586 INFO:teuthology.orchestra.run.smithi016.stdout:(17/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.4 MB/s | 315 kB 00:00 2020-03-11T22:30:03.606 INFO:teuthology.orchestra.run.smithi016.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 3.6 MB/s | 75 kB 00:00 2020-03-11T22:30:03.612 INFO:teuthology.orchestra.run.smithi016.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 31 MB/s | 179 kB 00:00 2020-03-11T22:30:03.658 INFO:teuthology.orchestra.run.smithi016.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 789 kB/s | 59 kB 00:00 2020-03-11T22:30:04.026 INFO:teuthology.orchestra.run.smithi016.stdout:(21/23): ceph-common-15.1.0-1941.g4acc112.el8.x 10 MB/s | 21 MB 00:02 2020-03-11T22:30:04.039 INFO:teuthology.orchestra.run.smithi016.stdout:(22/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 5.0 MB/s | 1.9 MB 00:00 2020-03-11T22:30:04.050 INFO:teuthology.orchestra.run.smithi016.stdout:(23/23): librados2-15.1.0-1941.g4acc112.el8.x86 8.2 MB/s | 3.6 MB 00:00 2020-03-11T22:30:04.052 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:04.053 INFO:teuthology.orchestra.run.smithi016.stdout:Total 24 MB/s | 51 MB 00:02 2020-03-11T22:30:04.059 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:30:04.106 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:30:04.107 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:30:04.459 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:30:04.460 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:30:05.548 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:30:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-11T22:30:06.129 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:30:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:30:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:30:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:07.447 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:07.846 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:07.918 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:08.116 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-11T22:30:08.238 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:30:08.922 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:30:09.062 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:30:09.151 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:30:09.264 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-11T22:30:09.458 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:30:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:30:09.831 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:30:10.004 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:30:10.060 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:30:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-11T22:30:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-11T22:30:10.580 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-11T22:30:10.731 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-11T22:30:10.879 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-11T22:30:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:15.428 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:30:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:30:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:32.758 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:32.758 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2020-03-11T22:30:32.758 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2020-03-11T22:30:32.758 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2020-03-11T22:30:32.758 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /run 2020-03-11T22:30:32.758 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2020-03-11T22:30:32.759 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2020-03-11T22:30:32.759 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2020-03-11T22:30:32.759 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2020-03-11T22:30:32.759 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2020-03-11T22:30:32.759 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:33.893 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:30:33.968 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:30:33.968 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:30:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:30:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:34.825 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:34.825 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-11T22:30:34.826 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-11T22:30:34.826 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:34.826 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:30:34.826 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:34.826 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:34.826 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-11T22:30:34.826 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-11T22:30:34.827 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-11T22:30:34.827 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-11T22:30:34.827 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:30:34.827 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:30:34.827 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:30:34.827 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-11T22:30:34.827 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-11T22:30:34.828 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-11T22:30:34.828 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-11T22:30:34.828 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-11T22:30:34.828 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-11T22:30:34.828 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-11T22:30:34.828 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-11T22:30:34.828 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:34.828 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-11T22:30:34.829 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-11T22:30:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:35.545 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:35.546 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2020-03-11T22:30:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:35.546 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:30:35.546 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.547 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-11T22:30:35.548 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-11T22:30:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-11T22:30:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-11T22:30:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-11T22:30:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-11T22:30:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-11T22:30:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: leveldb-1.21-2.el8.x86_64 2020-03-11T22:30:35.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:35.550 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:30:35.857 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-test 2020-03-11T22:30:36.295 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:36 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:30:36.689 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:30:36.690 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:36.690 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2020-03-11T22:30:36.690 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:36.690 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:30:36.690 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-11T22:30:36.691 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2020-03-11T22:30:36.691 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-11T22:30:36.691 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-11T22:30:36.691 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-11T22:30:36.691 INFO:teuthology.orchestra.run.smithi016.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-11T22:30:36.691 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-11T22:30:36.691 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:36.692 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:30:36.692 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:36.692 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2020-03-11T22:30:36.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:36.693 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 43 M 2020-03-11T22:30:36.693 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 184 M 2020-03-11T22:30:36.693 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:30:36.725 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.8 MB/s | 161 kB 00:00 2020-03-11T22:30:36.730 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.9 MB/s | 188 kB 00:00 2020-03-11T22:30:36.737 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 25 MB/s | 298 kB 00:00 2020-03-11T22:30:36.739 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 58 MB/s | 249 kB 00:00 2020-03-11T22:30:36.812 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 917 kB/s | 69 kB 00:00 2020-03-11T22:30:40.255 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 12 MB/s | 42 MB 00:03 2020-03-11T22:30:40.258 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:Total 12 MB/s | 43 MB 00:03 2020-03-11T22:30:40.266 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:30:40.290 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:30:40.290 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:30:40.566 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:30:40.566 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:30:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:30:41.211 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-11T22:30:41.398 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-11T22:30:41.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-11T22:30:41.716 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:30:41.796 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:30:46.127 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-11T22:30:46.338 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:30:46.548 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:30:46.548 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-11T22:30:46.548 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-11T22:30:46.548 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-11T22:30:46.549 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-11T22:30:46.549 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-11T22:30:46.989 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-11T22:30:46.989 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:46.989 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:30:46.990 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-11T22:30:46.990 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-11T22:30:46.990 INFO:teuthology.orchestra.run.smithi016.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-11T22:30:46.990 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:46.990 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:30:47.233 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph 2020-03-11T22:30:47.656 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:47 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:30:48.051 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:30:48.057 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: Package ArchVersion Repository Size 2020-03-11T22:30:48.057 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:48.057 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:30:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-11T22:30:48.058 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2020-03-11T22:30:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-11T22:30:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-11T22:30:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-11T22:30:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-11T22:30:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-11T22:30:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-11T22:30:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-11T22:30:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-11T22:30:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-11T22:30:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-11T22:30:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-11T22:30:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-11T22:30:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-11T22:30:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-11T22:30:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-11T22:30:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-11T22:30:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-11T22:30:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-11T22:30:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-11T22:30:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-11T22:30:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-11T22:30:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-11T22:30:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-11T22:30:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-11T22:30:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-11T22:30:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-11T22:30:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-11T22:30:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-11T22:30:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-11T22:30:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-11T22:30:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-11T22:30:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-11T22:30:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-11T22:30:48.063 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2020-03-11T22:30:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-11T22:30:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-11T22:30:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:48.064 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:30:48.064 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:30:48.064 INFO:teuthology.orchestra.run.smithi016.stdout:Install 36 Packages 2020-03-11T22:30:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:30:48.065 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 29 M 2020-03-11T22:30:48.065 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 105 M 2020-03-11T22:30:48.065 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:30:48.263 INFO:teuthology.orchestra.run.smithi016.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 32 kB/s | 6.4 kB 00:00 2020-03-11T22:30:48.435 INFO:teuthology.orchestra.run.smithi016.stdout:(2/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_6 3.8 MB/s | 1.4 MB 00:00 2020-03-11T22:30:48.574 INFO:teuthology.orchestra.run.smithi016.stdout:(3/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 4.0 MB/s | 2.0 MB 00:00 2020-03-11T22:30:48.638 INFO:teuthology.orchestra.run.smithi016.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 2.6 MB/s | 165 kB 00:00 2020-03-11T22:30:48.679 INFO:teuthology.orchestra.run.smithi016.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 5.2 MB/s | 214 kB 00:00 2020-03-11T22:30:48.782 INFO:teuthology.orchestra.run.smithi016.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2020-03-11T22:30:48.884 INFO:teuthology.orchestra.run.smithi016.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2020-03-11T22:30:48.945 INFO:teuthology.orchestra.run.smithi016.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 649 kB/s | 39 kB 00:00 2020-03-11T22:30:48.952 INFO:teuthology.orchestra.run.smithi016.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 7.6 MB/s | 54 kB 00:00 2020-03-11T22:30:49.063 INFO:teuthology.orchestra.run.smithi016.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 244 kB/s | 27 kB 00:00 2020-03-11T22:30:49.131 INFO:teuthology.orchestra.run.smithi016.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 6.7 MB/s | 457 kB 00:00 2020-03-11T22:30:49.229 INFO:teuthology.orchestra.run.smithi016.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 2.0 MB/s | 195 kB 00:00 2020-03-11T22:30:49.284 INFO:teuthology.orchestra.run.smithi016.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 629 kB/s | 34 kB 00:00 2020-03-11T22:30:49.347 INFO:teuthology.orchestra.run.smithi016.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 1.9 MB/s | 123 kB 00:00 2020-03-11T22:30:49.406 INFO:teuthology.orchestra.run.smithi016.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 2.9 MB/s | 176 kB 00:00 2020-03-11T22:30:49.456 INFO:teuthology.orchestra.run.smithi016.stdout:(16/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_ 3.7 MB/s | 4.4 MB 00:01 2020-03-11T22:30:49.615 INFO:teuthology.orchestra.run.smithi016.stdout:(17/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 699 kB/s | 145 kB 00:00 2020-03-11T22:30:49.636 INFO:teuthology.orchestra.run.smithi016.stdout:(18/36): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 385 kB 00:00 2020-03-11T22:30:49.723 INFO:teuthology.orchestra.run.smithi016.stdout:(19/36): python3-jaraco-6.2-6.el8.noarch.rpm 106 kB/s | 11 kB 00:00 2020-03-11T22:30:49.725 INFO:teuthology.orchestra.run.smithi016.stdout:(20/36): python3-jaraco-functools-2.0-4.el8.noa 204 kB/s | 18 kB 00:00 2020-03-11T22:30:49.783 INFO:teuthology.orchestra.run.smithi016.stdout:(21/36): python3-logutils-0.3.5-11.el8.noarch.r 823 kB/s | 49 kB 00:00 2020-03-11T22:30:49.822 INFO:teuthology.orchestra.run.smithi016.stdout:(22/36): python3-more-itertools-7.2.0-3.el8.noa 607 kB/s | 59 kB 00:00 2020-03-11T22:30:49.873 INFO:teuthology.orchestra.run.smithi016.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 323 kB/s | 16 kB 00:00 2020-03-11T22:30:49.877 INFO:teuthology.orchestra.run.smithi016.stdout:(24/36): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2020-03-11T22:30:49.881 INFO:teuthology.orchestra.run.smithi016.stdout:(25/36): python3-simplegeneric-0.8.1-17.el8.noa 2.2 MB/s | 19 kB 00:00 2020-03-11T22:30:49.890 INFO:teuthology.orchestra.run.smithi016.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 2.8 MB/s | 24 kB 00:00 2020-03-11T22:30:49.890 INFO:teuthology.orchestra.run.smithi016.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.4 MB/s | 30 kB 00:00 2020-03-11T22:30:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:(28/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 9.9 MB/s | 17 MB 00:01 2020-03-11T22:30:50.114 INFO:teuthology.orchestra.run.smithi016.stdout:(29/36): python3-trustme-0.5.2-4.el8.noarch.rpm 117 kB/s | 27 kB 00:00 2020-03-11T22:30:50.115 INFO:teuthology.orchestra.run.smithi016.stdout:(30/36): python3-zc-lockfile-2.0-2.el8.noarch.r 104 kB/s | 23 kB 00:00 2020-03-11T22:30:50.173 INFO:teuthology.orchestra.run.smithi016.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 711 kB/s | 40 kB 00:00 2020-03-11T22:30:50.177 INFO:teuthology.orchestra.run.smithi016.stdout:(32/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 696 kB/s | 44 kB 00:00 2020-03-11T22:30:50.207 INFO:teuthology.orchestra.run.smithi016.stdout:(33/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.0 MB/s | 185 kB 00:00 2020-03-11T22:30:50.223 INFO:teuthology.orchestra.run.smithi016.stdout:(34/36): python3-webtest-2.0.33-1.el8.noarch.rp 5.2 MB/s | 85 kB 00:00 2020-03-11T22:30:50.229 INFO:teuthology.orchestra.run.smithi016.stdout:(35/36): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2020-03-11T22:30:50.239 INFO:teuthology.orchestra.run.smithi016.stdout:(36/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.0 MB/s | 251 kB 00:00 2020-03-11T22:30:50.243 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:50.243 INFO:teuthology.orchestra.run.smithi016.stdout:Total 13 MB/s | 29 MB 00:02 2020-03-11T22:30:50.268 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:30:50.313 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:30:50.313 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:30:50.657 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:30:50.658 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:30:51.995 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:30:52.311 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-11T22:30:52.509 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-11T22:30:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-11T22:30:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-11T22:30:53.074 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-11T22:30:53.232 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-11T22:30:53.406 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-11T22:30:53.528 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-11T22:30:53.695 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-11T22:30:53.827 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-11T22:30:53.997 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-11T22:30:54.176 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-11T22:30:54.365 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-11T22:30:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-11T22:30:54.850 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-11T22:30:55.201 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-11T22:30:55.469 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-11T22:30:55.660 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-11T22:30:55.900 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-11T22:30:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-11T22:30:56.436 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-11T22:30:56.644 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-11T22:30:56.796 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-11T22:30:56.985 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-11T22:30:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-11T22:30:57.372 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-11T22:30:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-11T22:30:57.797 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-11T22:30:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-11T22:30:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-11T22:30:58.759 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-11T22:30:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:31:00.447 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:31:00.631 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:31:01.040 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:31:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:31:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:31:01.533 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:31:01.549 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:31:01.696 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:31:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:31:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-11T22:31:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-11T22:31:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-11T22:31:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-11T22:31:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-11T22:31:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-11T22:31:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-11T22:31:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-11T22:31:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-11T22:31:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-11T22:31:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-11T22:31:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-11T22:31:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-11T22:31:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-11T22:31:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-11T22:31:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-11T22:31:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-11T22:31:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-11T22:31:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-11T22:31:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-11T22:31:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-11T22:31:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-11T22:31:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-11T22:31:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-11T22:31:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-11T22:31:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-11T22:31:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-11T22:31:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-11T22:31:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-11T22:31:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-11T22:31:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-11T22:31:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-11T22:31:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-11T22:31:02.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-11T22:31:02.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-11T22:31:02.784 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-11T22:31:02.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:02.784 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:02.785 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:02.785 INFO:teuthology.orchestra.run.smithi016.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-11T22:31:02.785 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-11T22:31:02.785 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:02.785 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:02.785 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:02.785 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:02.786 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:02.786 INFO:teuthology.orchestra.run.smithi016.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-11T22:31:02.786 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-11T22:31:02.786 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-11T22:31:02.786 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-11T22:31:02.786 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-11T22:31:02.786 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-11T22:31:02.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-11T22:31:02.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-11T22:31:02.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-11T22:31:02.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-11T22:31:02.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-11T22:31:02.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-11T22:31:02.787 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-11T22:31:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-11T22:31:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-11T22:31:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-11T22:31:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-11T22:31:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-2.6-1.el8.noarch 2020-03-11T22:31:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-11T22:31:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-11T22:31:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-11T22:31:02.789 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-11T22:31:02.789 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-11T22:31:02.789 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-11T22:31:02.789 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-11T22:31:02.789 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-11T22:31:02.789 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-11T22:31:02.789 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-11T22:31:02.790 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:02.790 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:03.121 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base 2020-03-11T22:31:03.550 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:03 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:03.839 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:03.952 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:03.953 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:03.953 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:04.115 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install cephadm 2020-03-11T22:31:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:04 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:04.929 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2020-03-11T22:31:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:04.930 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:04.930 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-11T22:31:04.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:04.930 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:04.930 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:04.930 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2020-03-11T22:31:04.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:04.931 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 36 k 2020-03-11T22:31:04.931 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 129 k 2020-03-11T22:31:04.931 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:05.096 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 215 kB/s | 36 kB 00:00 2020-03-11T22:31:05.097 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:05.097 INFO:teuthology.orchestra.run.smithi016.stdout:Total 213 kB/s | 36 kB 00:00 2020-03-11T22:31:05.099 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:05.107 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:05.107 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:05.109 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:05.109 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:05.175 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:07.512 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:07.748 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:07.749 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:07.749 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:07.908 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-immutable-object-cache 2020-03-11T22:31:08.340 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:08 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:08.714 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:08.715 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:08.715 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2020-03-11T22:31:08.715 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:08.715 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:08.716 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-11T22:31:08.716 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:08.716 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:08.716 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:08.716 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2020-03-11T22:31:08.716 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:08.717 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 203 k 2020-03-11T22:31:08.717 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 743 k 2020-03-11T22:31:08.717 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:09.011 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 688 kB/s | 203 kB 00:00 2020-03-11T22:31:09.012 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:09.012 INFO:teuthology.orchestra.run.smithi016.stdout:Total 685 kB/s | 203 kB 00:00 2020-03-11T22:31:09.014 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:09.028 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:09.028 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:09.078 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:09.079 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:09.194 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:09.321 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:09.873 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:09.873 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:09.873 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:09.873 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:09.873 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:09.874 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:10.147 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr 2020-03-11T22:31:10.574 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:10 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:10.863 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:10.974 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:10.975 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:10.975 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:11.104 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-dashboard 2020-03-11T22:31:11.530 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:11 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:11.935 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:11.936 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:11.937 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:11.937 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-11T22:31:11.937 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2020-03-11T22:31:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-11T22:31:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-11T22:31:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-11T22:31:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-11T22:31:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:11.938 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:11.938 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:11.938 INFO:teuthology.orchestra.run.smithi016.stdout:Install 5 Packages 2020-03-11T22:31:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:11.939 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 4.2 M 2020-03-11T22:31:11.939 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 19 M 2020-03-11T22:31:11.939 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:11.970 INFO:teuthology.orchestra.run.smithi016.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2020-03-11T22:31:12.023 INFO:teuthology.orchestra.run.smithi016.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 635 kB/s | 34 kB 00:00 2020-03-11T22:31:12.049 INFO:teuthology.orchestra.run.smithi016.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 7.5 MB/s | 197 kB 00:00 2020-03-11T22:31:12.127 INFO:teuthology.orchestra.run.smithi016.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 119 kB/s | 22 kB 00:00 2020-03-11T22:31:12.423 INFO:teuthology.orchestra.run.smithi016.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 8.0 MB/s | 3.9 MB 00:00 2020-03-11T22:31:12.426 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:12.426 INFO:teuthology.orchestra.run.smithi016.stdout:Total 8.5 MB/s | 4.2 MB 00:00 2020-03-11T22:31:12.431 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:12.440 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:12.440 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:12.610 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:12.611 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:12.866 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:13.089 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-11T22:31:13.323 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-11T22:31:13.495 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:31:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-11T22:31:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:31:15.044 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:31:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-11T22:31:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-11T22:31:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:31:15.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-11T22:31:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-11T22:31:15.306 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:15.307 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-11T22:31:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-11T22:31:15.307 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-11T22:31:15.308 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:15.308 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:15.439 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-11T22:31:15.863 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:15 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:16.404 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:16.406 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:16.406 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:16.406 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-11T22:31:16.406 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2020-03-11T22:31:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-11T22:31:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-11T22:31:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-11T22:31:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-11T22:31:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-11T22:31:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-11T22:31:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-11T22:31:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:16.408 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:16.408 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:16.408 INFO:teuthology.orchestra.run.smithi016.stdout:Install 8 Packages 2020-03-11T22:31:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:16.409 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 37 M 2020-03-11T22:31:16.409 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 205 M 2020-03-11T22:31:16.409 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:16.535 INFO:teuthology.orchestra.run.smithi016.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 34 MB/s | 4.3 MB 00:00 2020-03-11T22:31:16.652 INFO:teuthology.orchestra.run.smithi016.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2020-03-11T22:31:16.713 INFO:teuthology.orchestra.run.smithi016.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 3.6 MB/s | 225 kB 00:00 2020-03-11T22:31:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:(4/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2020-03-11T22:31:16.846 INFO:teuthology.orchestra.run.smithi016.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 7.2 MB/s | 638 kB 00:00 2020-03-11T22:31:16.862 INFO:teuthology.orchestra.run.smithi016.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 10 MB/s | 169 kB 00:00 2020-03-11T22:31:17.158 INFO:teuthology.orchestra.run.smithi016.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 33 MB/s | 15 MB 00:00 2020-03-11T22:31:17.279 INFO:teuthology.orchestra.run.smithi016.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 9.9 MB/s | 8.6 MB 00:00 2020-03-11T22:31:17.281 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:17.281 INFO:teuthology.orchestra.run.smithi016.stdout:Total 42 MB/s | 37 MB 00:00 2020-03-11T22:31:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:17.314 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:17.314 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:17.692 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:17.693 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:18.649 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:31:18.914 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:31:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:31:19.647 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:31:19.803 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:31:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:31:21.612 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:31:21.819 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:31:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:31:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-11T22:31:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:31:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:31:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-11T22:31:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-11T22:31:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-11T22:31:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:31:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-11T22:31:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-11T22:31:26.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-11T22:31:26.046 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:26.046 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:26.046 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:26.046 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-11T22:31:26.047 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-11T22:31:26.047 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-11T22:31:26.047 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-11T22:31:26.047 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-11T22:31:26.047 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-11T22:31:26.047 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-11T22:31:26.047 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:26.048 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:26.182 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-rook 2020-03-11T22:31:26.614 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:26 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:27.027 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:27.029 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:27.029 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2020-03-11T22:31:27.029 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:27.030 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:27.030 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-11T22:31:27.030 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-noarch 24 k 2020-03-11T22:31:27.030 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2020-03-11T22:31:27.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-11T22:31:27.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-11T22:31:27.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-11T22:31:27.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-11T22:31:27.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-11T22:31:27.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-11T22:31:27.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-11T22:31:27.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-11T22:31:27.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-11T22:31:27.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-11T22:31:27.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-11T22:31:27.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-11T22:31:27.032 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:27.032 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:27.032 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:27.032 INFO:teuthology.orchestra.run.smithi016.stdout:Install 13 Packages 2020-03-11T22:31:27.033 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:27.033 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 1.9 M 2020-03-11T22:31:27.033 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 23 M 2020-03-11T22:31:27.033 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:27.066 INFO:teuthology.orchestra.run.smithi016.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 890 kB/s | 27 kB 00:00 2020-03-11T22:31:27.070 INFO:teuthology.orchestra.run.smithi016.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 15 MB/s | 126 kB 00:00 2020-03-11T22:31:27.070 INFO:teuthology.orchestra.run.smithi016.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 518 kB/s | 20 kB 00:00 2020-03-11T22:31:27.073 INFO:teuthology.orchestra.run.smithi016.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 34 MB/s | 110 kB 00:00 2020-03-11T22:31:27.075 INFO:teuthology.orchestra.run.smithi016.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 42 MB/s | 155 kB 00:00 2020-03-11T22:31:27.075 INFO:teuthology.orchestra.run.smithi016.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 26 MB/s | 43 kB 00:00 2020-03-11T22:31:27.122 INFO:teuthology.orchestra.run.smithi016.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 353 kB/s | 16 kB 00:00 2020-03-11T22:31:27.122 INFO:teuthology.orchestra.run.smithi016.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 696 kB/s | 33 kB 00:00 2020-03-11T22:31:27.153 INFO:teuthology.orchestra.run.smithi016.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.5 MB/s | 81 kB 00:00 2020-03-11T22:31:27.169 INFO:teuthology.orchestra.run.smithi016.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 4.4 MB/s | 73 kB 00:00 2020-03-11T22:31:27.185 INFO:teuthology.orchestra.run.smithi016.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.8 MB/s | 61 kB 00:00 2020-03-11T22:31:27.195 INFO:teuthology.orchestra.run.smithi016.stdout:(12/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 149 kB/s | 24 kB 00:00 2020-03-11T22:31:27.232 INFO:teuthology.orchestra.run.smithi016.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 10 MB/s | 1.1 MB 00:00 2020-03-11T22:31:27.233 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:27.234 INFO:teuthology.orchestra.run.smithi016.stdout:Total 9.2 MB/s | 1.9 MB 00:00 2020-03-11T22:31:27.241 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:27.257 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:27.257 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:27.329 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:27.329 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:27.851 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:28.141 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-11T22:31:28.377 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-11T22:31:28.859 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-11T22:31:29.035 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-11T22:31:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-11T22:31:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-11T22:31:29.759 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-11T22:31:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-11T22:31:31.072 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-11T22:31:31.309 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-11T22:31:31.501 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-11T22:31:31.709 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-11T22:31:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:31:32.116 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:31:32.116 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-11T22:31:32.116 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-11T22:31:32.116 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-11T22:31:32.116 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-11T22:31:32.117 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-11T22:31:32.117 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-11T22:31:32.117 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-11T22:31:32.117 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-11T22:31:32.117 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-11T22:31:32.117 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-11T22:31:32.117 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-11T22:31:32.118 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-11T22:31:32.453 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-11T22:31:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:32.454 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-11T22:31:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-11T22:31:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-11T22:31:32.454 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-11T22:31:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-11T22:31:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-11T22:31:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-11T22:31:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-11T22:31:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-11T22:31:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-11T22:31:32.455 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-11T22:31:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-11T22:31:32.456 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:32.456 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:32.618 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-cephadm 2020-03-11T22:31:33.043 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:33 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:33.451 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:33.452 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:33.453 INFO:teuthology.orchestra.run.smithi016.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:33.453 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:33.453 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:33.453 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-11T22:31:33.454 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2020-03-11T22:31:33.454 INFO:teuthology.orchestra.run.smithi016.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-11T22:31:33.454 INFO:teuthology.orchestra.run.smithi016.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-11T22:31:33.454 INFO:teuthology.orchestra.run.smithi016.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-11T22:31:33.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:33.455 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:33.455 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:33.455 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2020-03-11T22:31:33.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:33.456 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 461 k 2020-03-11T22:31:33.457 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 1.4 M 2020-03-11T22:31:33.457 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:33.507 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 911 kB/s | 48 kB 00:00 2020-03-11T22:31:33.520 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.3 MB/s | 356 kB 00:00 2020-03-11T22:31:33.564 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 374 kB/s | 21 kB 00:00 2020-03-11T22:31:33.618 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 221 kB/s | 36 kB 00:00 2020-03-11T22:31:33.620 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:33.620 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.7 MB/s | 461 kB 00:00 2020-03-11T22:31:33.624 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:33.631 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:33.632 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:33.662 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:33.662 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:33.841 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:34.083 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-11T22:31:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:31:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:31:35.459 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:31:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:31:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-11T22:31:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:31:35.699 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:31:35.961 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-11T22:31:35.962 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:35.962 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:35.962 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:35.962 INFO:teuthology.orchestra.run.smithi016.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-11T22:31:35.963 INFO:teuthology.orchestra.run.smithi016.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-11T22:31:35.963 INFO:teuthology.orchestra.run.smithi016.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-11T22:31:35.963 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:35.963 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:36.086 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-fuse 2020-03-11T22:31:36.514 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:36.898 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:36.899 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:36.899 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2020-03-11T22:31:36.899 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:36.899 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:36.899 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-11T22:31:36.900 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2020-03-11T22:31:36.900 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-11T22:31:36.900 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-11T22:31:36.900 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:36.900 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:36.900 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:36.900 INFO:teuthology.orchestra.run.smithi016.stdout:Install 3 Packages 2020-03-11T22:31:36.900 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:36.901 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 676 k 2020-03-11T22:31:36.901 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.1 M 2020-03-11T22:31:36.901 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:36.931 INFO:teuthology.orchestra.run.smithi016.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.7 MB/s | 82 kB 00:00 2020-03-11T22:31:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 621 kB/s | 21 kB 00:00 2020-03-11T22:31:37.237 INFO:teuthology.orchestra.run.smithi016.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.7 MB/s | 572 kB 00:00 2020-03-11T22:31:37.239 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:37.239 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.9 MB/s | 676 kB 00:00 2020-03-11T22:31:37.243 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:37.266 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:37.266 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:37.336 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:37.336 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-11T22:31:37.543 INFO:teuthology.orchestra.run.smithi016.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-11T22:31:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:31:38.141 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:31:38.433 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:31:38.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-11T22:31:38.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:31:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-11T22:31:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:38.701 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-11T22:31:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-11T22:31:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:38.701 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:38.962 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install librados-devel 2020-03-11T22:31:39.394 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:39.762 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:39.762 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2020-03-11T22:31:39.763 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:39.763 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:31:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:39.763 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:39.763 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:39.764 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2020-03-11T22:31:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:39.764 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 141 k 2020-03-11T22:31:39.764 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 487 k 2020-03-11T22:31:39.764 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:40.023 INFO:teuthology.orchestra.run.smithi016.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 543 kB/s | 141 kB 00:00 2020-03-11T22:31:40.023 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:40.024 INFO:teuthology.orchestra.run.smithi016.stdout:Total 540 kB/s | 141 kB 00:00 2020-03-11T22:31:40.025 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:40.033 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:40.033 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:40.087 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:40.087 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:41.024 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:41.233 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:41.527 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:41.527 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:41.527 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:41.527 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:41.527 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:41.528 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:41.812 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs2 2020-03-11T22:31:42.245 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:42.526 INFO:teuthology.orchestra.run.smithi016.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:42.640 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:42.641 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:42.641 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:42.775 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs-devel 2020-03-11T22:31:43.208 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:43.606 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:43.607 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2020-03-11T22:31:43.607 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:43.607 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-11T22:31:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:43.608 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:43.608 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:43.608 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2020-03-11T22:31:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:43.609 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 21 k 2020-03-11T22:31:43.609 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 76 k 2020-03-11T22:31:43.609 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:43.773 INFO:teuthology.orchestra.run.smithi016.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 129 kB/s | 21 kB 00:00 2020-03-11T22:31:43.774 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:43.774 INFO:teuthology.orchestra.run.smithi016.stdout:Total 128 kB/s | 21 kB 00:00 2020-03-11T22:31:43.776 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:43.780 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:43.780 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:43.783 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:43.783 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:43.834 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:44.427 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:44.428 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:44.602 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install librados2 2020-03-11T22:31:45.036 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:45.352 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:45.466 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:45.467 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:45.467 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:45.611 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install librbd1 2020-03-11T22:31:46.057 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:46.383 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:46.489 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:46.491 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:46.491 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:46.711 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rados 2020-03-11T22:31:47.142 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:47.427 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:47.539 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:47.540 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:47.540 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:47.669 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rgw 2020-03-11T22:31:48.096 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:48.386 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:48.496 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:48.498 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:48.627 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-cephfs 2020-03-11T22:31:49.054 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:49 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:49.335 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:49.447 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:49.448 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:49.448 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:49.574 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rbd 2020-03-11T22:31:50.003 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:50.289 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:50.402 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:50.403 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2020-03-11T22:31:50.403 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:50.527 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-fuse 2020-03-11T22:31:50.954 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:51.346 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:51.347 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2020-03-11T22:31:51.347 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:51.347 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:51.347 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-11T22:31:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:51.348 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:51.348 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:51.348 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2020-03-11T22:31:51.348 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:51.349 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 72 k 2020-03-11T22:31:51.349 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 230 k 2020-03-11T22:31:51.349 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:51.758 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 176 kB/s | 72 kB 00:00 2020-03-11T22:31:51.759 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:51.759 INFO:teuthology.orchestra.run.smithi016.stdout:Total 176 kB/s | 72 kB 00:00 2020-03-11T22:31:51.761 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:51.773 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:51.773 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:51.826 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:51.826 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:51.929 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:53.162 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:53.482 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:53.734 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:53.735 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:53.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:53.735 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:54.043 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror 2020-03-11T22:31:54.478 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:54 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:54.887 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:54.888 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:54.888 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2020-03-11T22:31:54.888 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:54.889 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-11T22:31:54.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:54.889 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:54.889 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:54.889 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2020-03-11T22:31:54.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:54.890 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.7 M 2020-03-11T22:31:54.890 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 12 M 2020-03-11T22:31:54.890 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:55.274 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 7.0 MB/s | 2.7 MB 00:00 2020-03-11T22:31:55.274 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:55.275 INFO:teuthology.orchestra.run.smithi016.stdout:Total 7.0 MB/s | 2.7 MB 00:00 2020-03-11T22:31:55.277 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:55.292 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:55.292 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:55.355 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:55.356 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:55.736 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:55.923 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:56.239 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:56.484 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:56.485 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:56.485 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:56.485 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:56.485 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:56.485 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:56.768 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-nbd 2020-03-11T22:31:57.200 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Mar 2020 10:30:00 PM UTC. 2020-03-11T22:31:57.647 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2020-03-11T22:31:57.647 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2020-03-11T22:31:57.648 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:57.648 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2020-03-11T22:31:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:31:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:57.648 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2020-03-11T22:31:57.648 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2020-03-11T22:31:57.649 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2020-03-11T22:31:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:57.649 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 141 k 2020-03-11T22:31:57.649 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 526 k 2020-03-11T22:31:57.650 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2020-03-11T22:31:57.908 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 545 kB/s | 141 kB 00:00 2020-03-11T22:31:57.908 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:57.908 INFO:teuthology.orchestra.run.smithi016.stdout:Total 543 kB/s | 141 kB 00:00 2020-03-11T22:31:57.910 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2020-03-11T22:31:57.922 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2020-03-11T22:31:57.923 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2020-03-11T22:31:57.975 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2020-03-11T22:31:57.976 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2020-03-11T22:31:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2020-03-11T22:31:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:59.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:59.276 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:59.276 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2020-03-11T22:31:59.276 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:59.276 INFO:teuthology.orchestra.run.smithi016.stdout: 2020-03-11T22:31:59.277 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2020-03-11T22:31:59.864 DEBUG:teuthology.parallel:result is None 2020-03-11T22:31:59.865 INFO:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2020-03-11T22:31:59.907 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS 2020-03-11T22:31:59.909 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:31:59.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:32:00.139 INFO:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-11T22:32:00.160 INFO:teuthology.orchestra.run.smithi016.stdout:15.1.0-1941.g4acc112.el8 2020-03-11T22:32:00.161 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-11T22:32:00.161 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-11T22:32:00.161 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-11T22:32:00.168 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-11T22:32:00.236 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-11T22:32:00.240 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-11T22:32:00.310 INFO:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-11T22:32:00.382 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-11T22:32:00.383 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-11T22:32:00.469 INFO:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-11T22:32:00.540 INFO:teuthology.run_tasks:Running task workunit... 2020-03-11T22:32:00.559 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:32:00.559 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-11T22:32:00.559 INFO:tasks.workunit:timeout=3h 2020-03-11T22:32:00.559 INFO:tasks.workunit:cleanup=True 2020-03-11T22:32:00.560 INFO:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:32:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-11T22:32:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:32:00.594 INFO:teuthology.orchestra.run.smithi016:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:32:00.649 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-11T22:32:00.649 INFO:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-11T22:32:00.706 INFO:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:32:00.817 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-11T22:33:36.825 INFO:tasks.workunit.client.0.smithi016.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-11T22:33:36.825 INFO:tasks.workunit.client.0.smithi016.stderr: 2020-03-11T22:33:36.825 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-11T22:33:36.826 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-11T22:33:36.826 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by performing another checkout. 2020-03-11T22:33:36.826 INFO:tasks.workunit.client.0.smithi016.stderr: 2020-03-11T22:33:36.826 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-11T22:33:36.826 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-11T22:33:36.826 INFO:tasks.workunit.client.0.smithi016.stderr: 2020-03-11T22:33:36.826 INFO:tasks.workunit.client.0.smithi016.stderr: git checkout -b 2020-03-11T22:33:36.827 INFO:tasks.workunit.client.0.smithi016.stderr: 2020-03-11T22:33:36.827 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-11T22:33:36.830 INFO:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-11T22:33:36.902 DEBUG:teuthology.orchestra.remote:smithi016:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-11T22:33:36.950 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-11T22:33:36.950 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-11T22:33:36.950 INFO:teuthology.orchestra.run.smithi016:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-11T22:33:36.979 INFO:tasks.workunit.client.0.smithi016.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:33:36.990 INFO:tasks.workunit.client.0.smithi016.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:33:36.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:33:36.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:33:36.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:33:36.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:33:36.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:33:36.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:33:36.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:33:36.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:33:36.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-11T22:33:36.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-11T22:33:36.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-11T22:33:36.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-11T22:33:36.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-11T22:33:36.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-11T22:33:36.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-11T22:33:36.994 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-11T22:33:36.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-11T22:33:36.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-11T22:33:36.995 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-11T22:33:36.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-11T22:33:36.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-11T22:33:36.995 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-11T22:33:36.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-11T22:33:36.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-11T22:33:36.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-11T22:33:36.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-11T22:33:36.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-11T22:33:36.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-11T22:33:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-11T22:33:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-11T22:33:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-11T22:33:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-11T22:33:37.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-11T22:33:37.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-11T22:33:37.000 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-11T22:33:37.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none ' 2020-03-11T22:33:37.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-11T22:33:37.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-11T22:33:37.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-11T22:33:37.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:37.004 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-11T22:33:37.004 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:33:37.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-11T22:33:37.007 INFO:tasks.workunit.client.0.smithi016.stderr:TEST_divergent_2 2020-03-11T22:33:37.007 INFO:tasks.workunit.client.0.smithi016.stderr:TEST_divergent_3 2020-03-11T22:33:37.008 INFO:tasks.workunit.client.0.smithi016.stderr:TEST_divergent_ec' 2020-03-11T22:33:37.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:33:37.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:33:37.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:33:37.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:33:37.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:33:37.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:33:37.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:33:37.010 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:33:37.010 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:33:37.010 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:33:37.010 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:33:37.011 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:33:37.011 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:33:37.011 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:33:37.011 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:33:37.011 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:33:37.015 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:33:37.015 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:33:37.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:33:37.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:33:37.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:33:37.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:33:37.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:33:37.017 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:33:37.017 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:33:37.017 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:33:37.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:33:37.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:37.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:37.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:33:37.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:33:37.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:33:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:33:37.019 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:33:37.019 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:37.019 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:37.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17411 2020-03-11T22:33:37.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:33:37.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:33:37.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:33:37.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:33:37.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:33:37.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-11T22:33:37.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-11T22:33:37.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-11T22:33:37.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-11T22:33:37.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-11T22:33:37.025 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-11T22:33:37.026 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-11T22:33:37.026 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-11T22:33:37.026 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:33:37.026 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:33:37.027 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-11T22:33:37.027 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:33:37.027 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:33:37.027 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:33:37.027 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:33:37.028 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:33:37.028 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:33:37.549 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:33:37.550 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:37.550 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:37.550 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:37.550 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:37.550 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:37.550 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:33:37.551 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:33:37.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:33:37.687 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:33:37.687 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:33:37.687 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:33:37.688 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:33:37.688 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:33:37.688 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:33:37.688 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:33:37.688 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:33:37.688 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:37.688 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:37.688 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:37.689 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-mon.a.asok 2020-03-11T22:33:37.689 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:33:37.689 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17411/ceph-mon.a.asok config get fsid 2020-03-11T22:33:37.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:33:37.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:33:37.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:33:37.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:33:37.784 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:33:37.784 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:33:37.785 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:33:37.785 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:33:37.785 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:37.785 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:37.785 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:37.785 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-mon.a.asok 2020-03-11T22:33:37.785 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:33:37.786 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17411/ceph-mon.a.asok config get mon_host 2020-03-11T22:33:37.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-11T22:33:37.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:33:37.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:33:37.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:33:37.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:33:37.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:33:37.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:33:38.930 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:33:38.930 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:38.930 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:38.930 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:38.931 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:38.931 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:38.931 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:33:38.931 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:33:38.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:33:38.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:33:38.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-11T22:33:38.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:33:38.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:33:38.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:33:38.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:33:38.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:33:38.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:38.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:33:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:33:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:33:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:38.964 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:33:38.964 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:38.964 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:38.964 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:38.964 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:38.964 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:38.964 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:33:38.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:33:38.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:38.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:33:38.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:33:38.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:38.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:38.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:38.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:38.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:38.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:33:38.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:33:38.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:33:38.966 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:33:38.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=be112234-e19a-4d23-85f2-fc6f4baa8e51 2020-03-11T22:33:38.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 be112234-e19a-4d23-85f2-fc6f4baa8e51' 2020-03-11T22:33:38.967 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:33:38.967 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 be112234-e19a-4d23-85f2-fc6f4baa8e51 2020-03-11T22:33:38.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBCZ2le/R1FOhAAbY9y+7/o372K9ZfMZjFK+g== 2020-03-11T22:33:38.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBCZ2le/R1FOhAAbY9y+7/o372K9ZfMZjFK+g=="}' 2020-03-11T22:33:38.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new be112234-e19a-4d23-85f2-fc6f4baa8e51 -i td/divergent-priors/0/new.json 2020-03-11T22:33:39.318 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:33:39.327 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:33:39.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBCZ2le/R1FOhAAbY9y+7/o372K9ZfMZjFK+g== --osd-uuid be112234-e19a-4d23-85f2-fc6f4baa8e51 2020-03-11T22:33:39.358 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:39.355+0000 7fe84382fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:39.360 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:39.357+0000 7fe84382fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:39.362 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:39.358+0000 7fe84382fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:39.362 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:39.359+0000 7fe84382fec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:39.363 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:39.359+0000 7fe84382fec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:39.363 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:39.359+0000 7fe84382fec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:39.451 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:39.447+0000 7fe84382fec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:33:42.480 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:33:42.481 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:33:42.482 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:33:42.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:33:42.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:33:42.737 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:33:42.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:42.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:33:42.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:33:42.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:33:42.738 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:33:42.766 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:42.762+0000 7f07fe746ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:42.779 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:42.775+0000 7f07fe746ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:42.780 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:42.777+0000 7f07fe746ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:43.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:33:43.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:43.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:33:43.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:43.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:43.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:43.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:43.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:43.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:33:43.141 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:33:43.300 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:43.296+0000 7f07fe746ec0 -1 Falling back to public interface 2020-03-11T22:33:43.508 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:44.040 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:44.037+0000 7f07fe746ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:33:44.510 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:44.510 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:44.510 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:44.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:44.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:33:44.511 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:33:44.942 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/17793,v1:127.0.0.1:6803/17793] [v2:127.0.0.1:6804/17793,v1:127.0.0.1:6805/17793] exists,up be112234-e19a-4d23-85f2-fc6f4baa8e51 2020-03-11T22:33:44.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:44.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:44.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:44.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:33:44.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-11T22:33:44.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:33:44.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:33:44.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:33:44.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:33:44.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:33:44.944 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:44.944 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:44.944 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:44.944 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:44.944 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:33:44.944 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:33:44.944 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:44.945 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:33:44.945 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:44.945 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:33:44.945 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:44.945 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:44.945 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:44.945 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:44.946 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:44.946 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:33:44.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:33:44.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:44.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:33:44.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:33:44.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:44.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:44.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:44.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:44.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:44.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:33:44.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:33:44.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:33:44.948 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:33:44.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=27627ef5-251f-4ae1-93b4-c3a2f74ee2f5 2020-03-11T22:33:44.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 27627ef5-251f-4ae1-93b4-c3a2f74ee2f5' 2020-03-11T22:33:44.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:33:44.949 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 27627ef5-251f-4ae1-93b4-c3a2f74ee2f5 2020-03-11T22:33:44.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBIZ2le0iobORAAm4QSoV71Fc9qCfz2NNLbGQ== 2020-03-11T22:33:44.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBIZ2le0iobORAAm4QSoV71Fc9qCfz2NNLbGQ=="}' 2020-03-11T22:33:44.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 27627ef5-251f-4ae1-93b4-c3a2f74ee2f5 -i td/divergent-priors/1/new.json 2020-03-11T22:33:45.515 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:33:45.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:33:45.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBIZ2le0iobORAAm4QSoV71Fc9qCfz2NNLbGQ== --osd-uuid 27627ef5-251f-4ae1-93b4-c3a2f74ee2f5 2020-03-11T22:33:45.556 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:45.553+0000 7fdcbed56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:45.569 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:45.566+0000 7fdcbed56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:45.570 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:45.568+0000 7fdcbed56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:45.571 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:45.568+0000 7fdcbed56ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:45.571 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:45.568+0000 7fdcbed56ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:45.571 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:45.568+0000 7fdcbed56ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:45.627 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:45.625+0000 7fdcbed56ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:33:48.794 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:33:48.794 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:33:48.796 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:33:48.796 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:33:48.796 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:33:49.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:33:49.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:49.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:33:49.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:33:49.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:33:49.289 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:33:49.315 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:49.313+0000 7f964f5f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:49.317 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:49.315+0000 7f964f5f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:49.319 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:49.317+0000 7f964f5f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:49.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:33:49.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:49.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:33:49.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:49.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:49.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:49.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:49.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:49.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:33:49.657 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:33:49.846 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:49.843+0000 7f964f5f4ec0 -1 Falling back to public interface 2020-03-11T22:33:50.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:50.594 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:50.591+0000 7f964f5f4ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:33:51.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:51.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:51.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:51.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:51.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:33:51.021 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:33:51.496 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/18203,v1:127.0.0.1:6811/18203] [v2:127.0.0.1:6812/18203,v1:127.0.0.1:6813/18203] exists,up 27627ef5-251f-4ae1-93b4-c3a2f74ee2f5 2020-03-11T22:33:51.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:51.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:51.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:51.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:33:51.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-11T22:33:51.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:33:51.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:33:51.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:33:51.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:33:51.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:33:51.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:51.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:51.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:51.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:51.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:33:51.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:33:51.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:51.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:33:51.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:51.499 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:33:51.500 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:51.500 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:51.500 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:51.500 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:51.500 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:33:51.500 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:33:51.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:33:51.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:51.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:33:51.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:33:51.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:51.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:51.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:51.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:51.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:51.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:33:51.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:33:51.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:33:51.503 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:33:51.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=058aee94-40ae-4d51-8e51-ca60b41dbcb5 2020-03-11T22:33:51.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 058aee94-40ae-4d51-8e51-ca60b41dbcb5' 2020-03-11T22:33:51.503 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:33:51.503 INFO:tasks.workunit.client.0.smithi016.stdout:add osd2 058aee94-40ae-4d51-8e51-ca60b41dbcb5 2020-03-11T22:33:51.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBPZ2leLkGpHhAABjiKsqncHDUYH9PHZdGt2A== 2020-03-11T22:33:51.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBPZ2leLkGpHhAABjiKsqncHDUYH9PHZdGt2A=="}' 2020-03-11T22:33:51.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 058aee94-40ae-4d51-8e51-ca60b41dbcb5 -i td/divergent-priors/2/new.json 2020-03-11T22:33:51.995 INFO:tasks.workunit.client.0.smithi016.stdout:2 2020-03-11T22:33:52.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:33:52.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBPZ2leLkGpHhAABjiKsqncHDUYH9PHZdGt2A== --osd-uuid 058aee94-40ae-4d51-8e51-ca60b41dbcb5 2020-03-11T22:33:52.035 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:52.033+0000 7fc930addec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:52.042 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:52.040+0000 7fc930addec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:52.044 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:52.042+0000 7fc930addec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:52.045 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:52.042+0000 7fc930addec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:33:52.045 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:52.042+0000 7fc930addec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:33:52.045 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:52.042+0000 7fc930addec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:33:52.126 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:52.124+0000 7fc930addec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:33:55.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:33:55.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:33:55.228 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd2 key to auth repository 2020-03-11T22:33:55.228 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:33:55.229 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:33:55.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:33:55.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:55.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:33:55.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:33:55.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:33:55.717 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:33:55.744 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:55.742+0000 7f1bf62a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:55.746 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:55.743+0000 7f1bf62a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:55.747 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:55.744+0000 7f1bf62a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:56.081 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:33:56.081 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:56.081 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:33:56.082 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:56.082 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:56.082 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:56.082 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:56.082 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:56.082 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:33:56.083 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:33:56.262 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:56.259+0000 7f1bf62a5ec0 -1 Falling back to public interface 2020-03-11T22:33:56.465 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:56.951 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:33:56.949+0000 7f1bf62a5ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:33:57.466 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:57.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:57.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:57.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:57.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:33:57.467 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:33:57.868 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/18566,v1:127.0.0.1:6819/18566] [v2:127.0.0.1:6820/18566,v1:127.0.0.1:6821/18566] exists,up 058aee94-40ae-4d51-8e51-ca60b41dbcb5 2020-03-11T22:33:57.868 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:57.868 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:57.868 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:57.868 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-11T22:33:58.521 INFO:tasks.workunit.client.0.smithi016.stderr:noout is set 2020-03-11T22:33:58.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-11T22:33:59.189 INFO:tasks.workunit.client.0.smithi016.stderr:noin is set 2020-03-11T22:33:59.197 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-11T22:33:59.916 INFO:tasks.workunit.client.0.smithi016.stderr:nodown is set 2020-03-11T22:33:59.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-11T22:33:59.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:34:00.641 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' already exists 2020-03-11T22:34:00.649 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:34:01.650 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-11T22:34:02.778 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 size to 3 2020-03-11T22:34:02.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-11T22:34:03.686 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 min_size to 2 2020-03-11T22:34:03.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-11T22:34:03.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:03.695 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:04.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:04.196 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:34:04.196 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:34:04.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:04.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:04.197 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:04.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:34:04.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:34:04.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:04.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:04.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:34:04.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:34:04.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:04.472 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:04.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:34:04.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:34:04.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:04.610 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:34:04.610 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:04.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:04.612 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:34:04.612 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:04.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:34:04.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:34:04.613 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:34:04.613 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:04.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:34:04.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:05.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:05.980 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:06.356 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:34:06.357 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:06.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:34:06.358 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:06.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:06.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:34:06.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:06.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:34:06.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:34:06.361 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:34:06.361 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:06.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:34:06.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:06.750 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:34:06.750 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:06.750 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:06.752 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:34:06.752 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:06.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:34:06.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:34:06.753 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:34:06.754 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:07.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:34:07.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-11T22:34:07.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:34:07.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:34:07.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:34:07.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:34:07.156 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:34:07.157 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:34:07.157 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:34:07.157 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:34:07.157 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:34:07.157 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:34:07.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:34:07.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:34:07.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:34:07.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:07.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:07.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:07.613 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:34:07.613 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:34:07.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:07.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:07.614 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:08.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:34:08.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:34:08.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:08.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:08.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:34:08.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:34:08.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:08.189 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:08.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:34:08.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-03-11T22:34:08.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:08.325 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:34:08.325 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:08.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:08.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:34:08.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:08.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:34:08.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:34:08.329 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:34:08.329 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:08.764 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:34:08.764 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:09.765 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:09.766 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:10.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:34:10.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:10.144 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:34:10.144 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:10.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:10.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:34:10.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:10.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:34:10.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:34:10.147 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:34:10.148 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:10.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:34:10.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:10.536 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:34:10.536 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:10.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:10.538 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:34:10.538 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:10.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:34:10.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:34:10.539 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:34:10.540 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:10.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:34:10.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:10.906 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:10.906 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:11.401 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:11.401 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:11.402 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:11.402 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:11.402 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:11.402 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:11.403 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:11.403 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:11.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:34:11.757 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:11.757 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:11.757 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:12.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:34:12.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:34:12.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:34:12.233 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-11T22:34:12.233 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-11T22:34:12.586 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:34:12.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-11T22:34:12.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-11T22:34:12.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:12.595 INFO:tasks.workunit.client.0.smithi016.stdout:primary and soon to be divergent is 1 2020-03-11T22:34:12.939 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:12.939 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:34:01.956662+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:34:12.939 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:34:12.939 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:12.940 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:34:12.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-11T22:34:12.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:12.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-11T22:34:12.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-11T22:34:12.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:12.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-11T22:34:12.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-11T22:34:12.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:12.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-11T22:34:12.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-11T22:34:12.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-11T22:34:12.952 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-11T22:34:12.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:12.953 INFO:tasks.workunit.client.0.smithi016.stdout:writing initial objects 2020-03-11T22:34:12.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-11T22:34:13.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-11T22:34:13.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-11T22:34:13.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-11T22:34:13.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-11T22:34:13.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-11T22:34:13.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-11T22:34:13.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-11T22:34:13.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-11T22:34:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-11T22:34:13.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-11T22:34:13.725 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.726 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-11T22:34:13.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-11T22:34:13.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-11T22:34:13.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:13.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-11T22:34:14.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-11T22:34:14.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-11T22:34:14.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-11T22:34:14.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-11T22:34:14.368 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.368 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-11T22:34:14.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-11T22:34:14.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-11T22:34:14.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-11T22:34:14.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-11T22:34:14.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-11T22:34:14.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.739 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-11T22:34:14.839 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.839 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-11T22:34:14.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-11T22:34:14.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:14.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-11T22:34:15.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-11T22:34:15.097 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.098 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-11T22:34:15.223 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.223 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-11T22:34:15.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.277 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-11T22:34:15.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-11T22:34:15.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-11T22:34:15.443 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.443 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-11T22:34:15.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-11T22:34:15.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-11T22:34:15.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-11T22:34:15.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-11T22:34:15.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-11T22:34:15.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-11T22:34:15.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-11T22:34:15.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:15.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-11T22:34:16.043 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-11T22:34:16.092 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.093 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-11T22:34:16.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-11T22:34:16.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-11T22:34:16.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-11T22:34:16.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-11T22:34:16.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-11T22:34:16.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-11T22:34:16.577 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.577 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-11T22:34:16.635 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.635 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-11T22:34:16.684 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.684 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-11T22:34:16.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-11T22:34:16.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-11T22:34:16.910 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.911 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-11T22:34:16.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:16.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-11T22:34:17.010 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.010 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-11T22:34:17.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-11T22:34:17.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.327 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-11T22:34:17.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-11T22:34:17.459 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.460 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-11T22:34:17.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-11T22:34:17.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.560 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-11T22:34:17.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-11T22:34:17.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-11T22:34:17.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-11T22:34:17.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:17.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-11T22:34:18.069 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-11T22:34:18.178 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.178 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-11T22:34:18.303 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.303 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-11T22:34:18.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-11T22:34:18.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-11T22:34:18.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-11T22:34:18.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-11T22:34:18.568 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-11T22:34:18.662 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.662 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-11T22:34:18.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-11T22:34:18.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-11T22:34:18.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-11T22:34:18.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-11T22:34:18.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:18.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-11T22:34:19.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-11T22:34:19.228 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.228 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-11T22:34:19.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-11T22:34:19.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-11T22:34:19.403 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.403 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-11T22:34:19.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-11T22:34:19.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-11T22:34:19.619 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-11T22:34:19.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-11T22:34:19.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-11T22:34:19.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-11T22:34:19.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-11T22:34:19.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:19.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-11T22:34:20.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-11T22:34:20.094 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-11T22:34:20.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:34:20.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:34:20.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-11T22:34:20.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:34:20.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:34:20.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:34:20.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:34:20.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:34:20.250 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:34:20.250 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:34:20.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:34:20.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:34:20.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:34:20.306 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:34:20.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:34:20.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:34:20.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:20.307 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:20.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:20.815 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:34:20.815 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:34:20.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:20.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:20.815 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:20.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-11T22:34:20.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-11T22:34:20.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:20.954 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:21.088 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:34:21.089 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2020-03-11T22:34:21.089 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:21.089 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:21.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:34:21.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451' 2020-03-11T22:34:21.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:21.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-11T22:34:21.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:21.228 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:21.229 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-11T22:34:21.229 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:21.230 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-11T22:34:21.230 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-11T22:34:21.230 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836494 2020-03-11T22:34:21.231 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:21.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836494 2020-03-11T22:34:21.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:21.600 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:34:21.600 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:21.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:21.602 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:34:21.602 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:21.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:34:21.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:34:21.603 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:34:21.604 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:21.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672970 -lt 42949672972 2020-03-11T22:34:21.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:23.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:23.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:23.382 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672972 2020-03-11T22:34:23.382 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:23.383 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:34:23.383 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:23.384 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:23.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:34:23.386 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:23.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:34:23.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:34:23.387 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:34:23.387 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:23.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:34:23.759 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:23.760 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:23.760 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:24.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:24.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:24.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:24.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:24.228 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:24.228 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:24.228 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:24.228 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:24.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:34:24.581 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:24.582 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:24.582 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:25.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:34:25.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:34:25.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:34:25.055 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-11T22:34:25.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:34:25.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:34:25.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:34:25.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:34:25.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-11T22:34:25.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-11T22:34:25.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:25.426 INFO:tasks.workunit.client.0.smithi016.stdout:blackholing osds 0 2 2020-03-11T22:34:25.765 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:25.765 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:34:01.956662+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:34:25.765 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:34:25.766 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:25.767 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:34:25.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:25.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-11T22:34:25.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:34:25.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:34:25.777 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:25.777 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:25.777 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:34:25.777 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.0.asok 2020-03-11T22:34:25.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:25.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:34:25.868 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:34:25.868 INFO:tasks.workunit.client.0.smithi016.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:34:25.868 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:34:25.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:25.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-11T22:34:25.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:34:25.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:34:25.877 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:25.877 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:25.877 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:34:25.878 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.2.asok 2020-03-11T22:34:25.878 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:25.878 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:34:25.968 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:34:25.969 INFO:tasks.workunit.client.0.smithi016.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:34:25.969 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:34:25.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-11T22:34:25.976 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-11T22:34:25.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-11T22:34:25.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-11T22:34:25.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-11T22:34:25.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-11T22:34:25.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:34:25.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-11T22:34:25.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-11T22:34:25.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-11T22:34:25.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-11T22:34:25.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-11T22:34:25.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-11T22:34:25.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-11T22:34:25.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-11T22:34:25.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-11T22:34:25.982 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-11T22:34:25.982 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-11T22:34:25.982 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-11T22:34:25.982 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-11T22:34:25.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-11T22:34:25.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-11T22:34:25.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-11T22:34:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-11T22:34:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-11T22:34:25.983 INFO:tasks.workunit.client.0.smithi016.stdout:writing divergent object 2020-03-11T22:34:25.983 INFO:tasks.workunit.client.0.smithi016.stdout:create missing divergent object 2020-03-11T22:34:26.349 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-11T22:34:26.349 INFO:tasks.workunit.client.0.smithi016.stderr:0 2020-03-11T22:34:26.349 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:34:26.349 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-11T22:34:26.349 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-11T22:34:26.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-11T22:34:26.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-11T22:34:26.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-11T22:34:26.352 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-11T22:34:26.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-11T22:34:26.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-11T22:34:26.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-11T22:34:26.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-11T22:34:26.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-11T22:34:26.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-11T22:34:26.353 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-11T22:34:26.353 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-11T22:34:26.354 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:34:26.354 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:34:26.354 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:26.354 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:26.354 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:34:26.354 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.1.asok 2020-03-11T22:34:26.355 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.17411/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-11T22:34:26.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-11T22:34:26.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-11T22:34:26.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-11T22:34:26.453 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-11T22:34:26.453 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:34:26.453 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:34:26.453 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:26.454 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:26.454 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:34:26.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.1.asok 2020-03-11T22:34:26.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-11T22:34:26.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.1.asok injectdataerr test existing_99 2020-03-11T22:34:26.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-11T22:34:26.554 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-11T22:34:36.554 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-11T22:34:36.580 INFO:tasks.workunit.client.0.smithi016.stderr:rados: no process found 2020-03-11T22:34:36.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-11T22:34:36.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:36.580 INFO:tasks.workunit.client.0.smithi016.stdout:killing all the osds 2020-03-11T22:34:36.930 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:36.931 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-11T22:34:26.906858+0000 28'101 28:120 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:34:36.931 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:34:36.931 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:36.932 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:34:36.940 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:34:36.941 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:34:36.942 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:34:36.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:34:36.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:34:36.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:34:37.048 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:34:37.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:37.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-11T22:34:37.539 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already down. 2020-03-11T22:34:37.547 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:37.548 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-11T22:34:38.038 INFO:tasks.workunit.client.0.smithi016.stderr:osd.1 is already down. 2020-03-11T22:34:38.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:38.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-11T22:34:38.597 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already down. 2020-03-11T22:34:38.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:38.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-11T22:34:39.106 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already out. 2020-03-11T22:34:39.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:39.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-11T22:34:39.640 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already out. 2020-03-11T22:34:39.643 INFO:tasks.workunit.client.0.smithi016.stdout:bringing up non_divergent 0 2 2020-03-11T22:34:39.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-11T22:34:39.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:39.981 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:39.981 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-11T22:34:26.906858+0000 28'101 28:120 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:34:39.981 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:34:39.981 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:39.982 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:34:39.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:39.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-11T22:34:39.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:34:39.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:34:39.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:34:39.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:34:39.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:34:39.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:39.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:39.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:39.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:40.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:34:40.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:34:40.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:40.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:34:40.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:40.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:34:40.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:40.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:40.002 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:40.002 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:40.002 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:34:40.002 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:34:40.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:34:40.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:40.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:40.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:40.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:40.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:40.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:40.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:34:40.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:34:40.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:34:40.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:34:40.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:40.005 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:34:40.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:34:40.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:34:40.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:34:40.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:34:40.006 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:34:40.021 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:40.017+0000 7f5a6926eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:40.023 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:40.019+0000 7f5a6926eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:40.024 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:40.020+0000 7f5a6926eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:40.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:34:40.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:40.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:34:40.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:40.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:40.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:40.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:40.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:40.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:34:40.363 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:34:40.540 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:40.536+0000 7f5a6926eec0 -1 Falling back to public interface 2020-03-11T22:34:40.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:34:41.613 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:41.609+0000 7f5a6926eec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:34:41.723 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:34:41.724 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:41.724 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:34:41.724 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:41.724 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:34:41.724 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:34:42.108 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/22585,v1:127.0.0.1:6803/22585] [v2:127.0.0.1:6804/22585,v1:127.0.0.1:6805/22585] exists,up be112234-e19a-4d23-85f2-fc6f4baa8e51 2020-03-11T22:34:42.108 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:34:42.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:34:42.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:34:42.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:34:42.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:34:42.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:42.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:42.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:42.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:42.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:34:42.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:34:42.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:42.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:34:42.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:42.111 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:34:42.111 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:42.111 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:42.111 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:42.112 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:42.112 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:34:42.112 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:34:42.112 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:34:42.112 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:42.112 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:42.112 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:42.112 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:42.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:42.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:42.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:34:42.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:34:42.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:34:42.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:34:42.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:42.114 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:34:42.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:34:42.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:34:42.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:34:42.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:34:42.135 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:42.133+0000 7f524dea9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:42.137 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:42.134+0000 7f524dea9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:42.138 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:42.134+0000 7f524dea9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:42.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:34:42.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:42.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:34:42.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:42.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:42.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:42.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:42.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:42.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:42.473 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:34:42.654 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:42.650+0000 7f524dea9ec0 -1 Falling back to public interface 2020-03-11T22:34:42.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:34:43.627 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:34:43.624+0000 7f524dea9ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:34:43.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:34:43.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:43.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:34:43.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:43.858 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:34:43.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:44.233 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/22779,v1:127.0.0.1:6811/22779] [v2:127.0.0.1:6812/22779,v1:127.0.0.1:6813/22779] exists,up 058aee94-40ae-4d51-8e51-ca60b41dbcb5 2020-03-11T22:34:44.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:44.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:44.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:44.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:44.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-11T22:34:44.768 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already in. 2020-03-11T22:34:44.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:44.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-11T22:34:45.410 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already in. 2020-03-11T22:34:45.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:34:45.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-11T22:34:45.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:34:45.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:34:45.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:34:45.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:34:45.420 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:34:45.420 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:34:45.420 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:34:45.421 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:34:45.421 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:34:45.421 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:34:45.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:34:45.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:34:45.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:34:45.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:45.478 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:45.845 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:45.845 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:34:45.845 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:34:45.845 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:45.845 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:45.846 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:45.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:34:45.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:34:45.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:45.984 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:46.079 INFO:tasks.workunit.client.0.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:34:46.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:34:46.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:34:46.088 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:46.088 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:46.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:34:46.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:34:46.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:46.225 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:34:46.225 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:46.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:46.226 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:34:46.226 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:46.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:34:46.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:34:46.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:46.228 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:34:46.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:34:46.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:47.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:47.593 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:47.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-11T22:34:47.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:47.964 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:34:47.965 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:47.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:47.966 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:34:47.966 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:47.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:34:47.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:34:47.967 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:47.967 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 2020-03-11T22:34:48.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 2020-03-11T22:34:48.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672976: unary operator expected 2020-03-11T22:34:48.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:48.362 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:34:48.363 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:48.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:48.363 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:34:48.364 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:48.364 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:34:48.364 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:34:48.365 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:48.365 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:34:48.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:34:48.773 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:48.773 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:48.773 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:49.277 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:49.277 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:49.277 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:49.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:49.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:49.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:49.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:49.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:49.628 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:49.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:49.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:49.629 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:50.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:50.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:34:50.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:34:50.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:34:50.100 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:50.100 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:34:50.200 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:50.200 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:50.200 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:50.201 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:50.201 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:50.201 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:50.201 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:50.201 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:50.551 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:50.551 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:50.552 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:50.552 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:51.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:51.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:51.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:51.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:51.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:51.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:51.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:51.025 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:51.025 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:51.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:51.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:51.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:34:51.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:51.504 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:34:51.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:51.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:51.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:51.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:51.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:51.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:51.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:51.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:52.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:52.054 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:52.054 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:52.054 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:52.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:52.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:52.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:52.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:52.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:52.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:52.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:52.525 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:52.525 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:53.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:53.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:53.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:34:53.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:53.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:34:53.403 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:53.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:53.405 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:53.405 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:53.405 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:53.405 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:53.405 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:53.406 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:53.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:53.755 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:53.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:53.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:54.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:54.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:54.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:54.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:54.225 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:54.225 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:54.225 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:54.225 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:54.225 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:54.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:54.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:54.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:34:54.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:54.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:34:55.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:55.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:55.496 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:55.496 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:55.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:55.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:55.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:55.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:55.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:55.860 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:55.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:55.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:56.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:56.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:56.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:56.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:56.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:56.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:56.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:56.333 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:56.333 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:56.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:56.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:56.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:34:56.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:56.807 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:34:58.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:58.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:58.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:58.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:58.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:58.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:58.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:58.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:58.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:58.772 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:58.772 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:58.773 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:59.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:59.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:59.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:59.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:59.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:59.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:59.246 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:59.246 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:59.246 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:59.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:59.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:59.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:34:59.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:59.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:35:02.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:02.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:02.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:02.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:02.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:02.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:02.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:02.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:03.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:03.284 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:03.285 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:03.285 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:03.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:03.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:03.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:03.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:03.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:03.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:03.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:03.769 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:03.769 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:04.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:04.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:04.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:35:04.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:04.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:35:10.641 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:10.641 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:10.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:10.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:10.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:10.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:10.643 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:10.643 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:11.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:11.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:11.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:11.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:11.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:11.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:11.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:11.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:11.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:11.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:11.483 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:11.483 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:11.483 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:11.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:11.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:11.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:35:11.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:11.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:35:19.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:19.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:19.255 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:19.255 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:19.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:19.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:19.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:19.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:19.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:19.616 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:19.617 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:19.617 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:20.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:20.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:20.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:20.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:20.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:20.088 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:20.088 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:20.088 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:20.088 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:20.561 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:20.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:20.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:35:20.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:35:21.032 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:35:21.032 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_fingerprint": "4622e476-56cc-4210-8100-51c89d6ab2a6", 2020-03-11T22:35:21.032 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:35:21.032 INFO:tasks.workunit.client.0.smithi016.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:35:21.033 INFO:tasks.workunit.client.0.smithi016.stdout: "timestamp": "2020-03-11T22:35:21.028887+0000", 2020-03-11T22:35:21.033 INFO:tasks.workunit.client.0.smithi016.stdout: "tag": "", 2020-03-11T22:35:21.033 INFO:tasks.workunit.client.0.smithi016.stdout: "health": { 2020-03-11T22:35:21.033 INFO:tasks.workunit.client.0.smithi016.stdout: "status": "HEALTH_WARN", 2020-03-11T22:35:21.033 INFO:tasks.workunit.client.0.smithi016.stdout: "checks": { 2020-03-11T22:35:21.033 INFO:tasks.workunit.client.0.smithi016.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:35:21.033 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:21.034 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:35:21.034 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:35:21.034 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 17 2020-03-11T22:35:21.034 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.034 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [], 2020-03-11T22:35:21.034 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:35:21.034 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: "OSD_DOWN": { 2020-03-11T22:35:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:35:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 osds down", 2020-03-11T22:35:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:35:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:35:21.036 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.036 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "osd.1 (root=default,host=smithi016) is down" 2020-03-11T22:35:21.036 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.036 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.036 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:35:21.036 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.036 INFO:tasks.workunit.client.0.smithi016.stdout: "PG_DEGRADED": { 2020-03-11T22:35:21.036 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:21.037 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:35:21.037 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:35:21.037 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:35:21.037 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.037 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:35:21.037 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.037 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:35:21.038 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.038 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.038 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:35:21.038 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.038 INFO:tasks.workunit.client.0.smithi016.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:35:21.038 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:21.038 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:35:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:35:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:35:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:35:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:35:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.039 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.040 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:35:21.040 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.040 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.040 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:35:21.040 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.040 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.040 INFO:tasks.workunit.client.0.smithi016.stdout: "mutes": [] 2020-03-11T22:35:21.041 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.041 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_first_committed": 1, 2020-03-11T22:35:21.041 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_last_committed": 1, 2020-03-11T22:35:21.041 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap": { 2020-03-11T22:35:21.041 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:35:21.041 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:35:21.041 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:33:37.071636Z", 2020-03-11T22:35:21.042 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:33:37.071636Z", 2020-03-11T22:35:21.042 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release": 15, 2020-03-11T22:35:21.042 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:35:21.042 INFO:tasks.workunit.client.0.smithi016.stdout: "features": { 2020-03-11T22:35:21.042 INFO:tasks.workunit.client.0.smithi016.stdout: "persistent": [ 2020-03-11T22:35:21.042 INFO:tasks.workunit.client.0.smithi016.stdout: "kraken", 2020-03-11T22:35:21.043 INFO:tasks.workunit.client.0.smithi016.stdout: "luminous", 2020-03-11T22:35:21.043 INFO:tasks.workunit.client.0.smithi016.stdout: "mimic", 2020-03-11T22:35:21.043 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap-prune", 2020-03-11T22:35:21.043 INFO:tasks.workunit.client.0.smithi016.stdout: "nautilus", 2020-03-11T22:35:21.043 INFO:tasks.workunit.client.0.smithi016.stdout: "octopus" 2020-03-11T22:35:21.043 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.043 INFO:tasks.workunit.client.0.smithi016.stdout: "optional": [] 2020-03-11T22:35:21.044 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.044 INFO:tasks.workunit.client.0.smithi016.stdout: "mons": [ 2020-03-11T22:35:21.044 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.044 INFO:tasks.workunit.client.0.smithi016.stdout: "rank": 0, 2020-03-11T22:35:21.044 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "a", 2020-03-11T22:35:21.044 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:35:21.045 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.045 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.045 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.045 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:35:21.045 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 0 2020-03-11T22:35:21.045 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:35:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:35:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 0, 2020-03-11T22:35:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 0 2020-03-11T22:35:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.046 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.047 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.047 INFO:tasks.workunit.client.0.smithi016.stdout: "quorum": [ 2020-03-11T22:35:21.047 INFO:tasks.workunit.client.0.smithi016.stdout: 0 2020-03-11T22:35:21.047 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.047 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap": { 2020-03-11T22:35:21.047 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 40, 2020-03-11T22:35:21.047 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:35:21.048 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:33:37.920784+0000", 2020-03-11T22:35:21.048 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:34:45.481770+0000", 2020-03-11T22:35:21.048 INFO:tasks.workunit.client.0.smithi016.stdout: "last_up_change": "2020-03-11T22:34:43.701557+0000", 2020-03-11T22:35:21.048 INFO:tasks.workunit.client.0.smithi016.stdout: "last_in_change": "2020-03-11T22:34:45.112664+0000", 2020-03-11T22:35:21.048 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:35:21.048 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_num": 5800384, 2020-03-11T22:35:21.048 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_set": [ 2020-03-11T22:35:21.048 INFO:tasks.workunit.client.0.smithi016.stdout: "nodown", 2020-03-11T22:35:21.049 INFO:tasks.workunit.client.0.smithi016.stdout: "noin", 2020-03-11T22:35:21.049 INFO:tasks.workunit.client.0.smithi016.stdout: "noout", 2020-03-11T22:35:21.049 INFO:tasks.workunit.client.0.smithi016.stdout: "pglog_hardlimit", 2020-03-11T22:35:21.049 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snapdirs", 2020-03-11T22:35:21.049 INFO:tasks.workunit.client.0.smithi016.stdout: "recovery_deletes", 2020-03-11T22:35:21.049 INFO:tasks.workunit.client.0.smithi016.stdout: "sortbitwise" 2020-03-11T22:35:21.049 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.050 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_version": 7, 2020-03-11T22:35:21.050 INFO:tasks.workunit.client.0.smithi016.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:35:21.050 INFO:tasks.workunit.client.0.smithi016.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:35:21.050 INFO:tasks.workunit.client.0.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:35:21.050 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_snapshot": "", 2020-03-11T22:35:21.050 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_max": 1, 2020-03-11T22:35:21.050 INFO:tasks.workunit.client.0.smithi016.stdout: "max_osd": 3, 2020-03-11T22:35:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:35:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: "min_compat_client": "jewel", 2020-03-11T22:35:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: "require_osd_release": "octopus", 2020-03-11T22:35:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: "pools": [ 2020-03-11T22:35:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:35:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_name": "test", 2020-03-11T22:35:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: "create_time": "2020-03-11T22:34:00.194631+0000", 2020-03-11T22:35:21.052 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": 1, 2020-03-11T22:35:21.052 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_names": "hashpspool", 2020-03-11T22:35:21.052 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:35:21.052 INFO:tasks.workunit.client.0.smithi016.stdout: "size": 3, 2020-03-11T22:35:21.052 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 2, 2020-03-11T22:35:21.052 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_rule": 0, 2020-03-11T22:35:21.052 INFO:tasks.workunit.client.0.smithi016.stdout: "object_hash": 2, 2020-03-11T22:35:21.053 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:35:21.053 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num": 1, 2020-03-11T22:35:21.053 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num": 1, 2020-03-11T22:35:21.053 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num_target": 1, 2020-03-11T22:35:21.053 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_target": 1, 2020-03-11T22:35:21.053 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_pending": 1, 2020-03-11T22:35:21.053 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pg_merge_meta": { 2020-03-11T22:35:21.053 INFO:tasks.workunit.client.0.smithi016.stdout: "source_pgid": "0.0", 2020-03-11T22:35:21.054 INFO:tasks.workunit.client.0.smithi016.stdout: "ready_epoch": 0, 2020-03-11T22:35:21.054 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_started": 0, 2020-03-11T22:35:21.054 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_clean": 0, 2020-03-11T22:35:21.054 INFO:tasks.workunit.client.0.smithi016.stdout: "source_version": "0'0", 2020-03-11T22:35:21.054 INFO:tasks.workunit.client.0.smithi016.stdout: "target_version": "0'0" 2020-03-11T22:35:21.054 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.054 INFO:tasks.workunit.client.0.smithi016.stdout: "last_change": "28", 2020-03-11T22:35:21.055 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend": "0", 2020-03-11T22:35:21.055 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:35:21.055 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:35:21.055 INFO:tasks.workunit.client.0.smithi016.stdout: "auid": 0, 2020-03-11T22:35:21.055 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:35:21.055 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_seq": 0, 2020-03-11T22:35:21.055 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_epoch": 0, 2020-03-11T22:35:21.055 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_snaps": [], 2020-03-11T22:35:21.056 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps": "[]", 2020-03-11T22:35:21.056 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_bytes": 0, 2020-03-11T22:35:21.056 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_objects": 0, 2020-03-11T22:35:21.056 INFO:tasks.workunit.client.0.smithi016.stdout: "tiers": [], 2020-03-11T22:35:21.056 INFO:tasks.workunit.client.0.smithi016.stdout: "tier_of": -1, 2020-03-11T22:35:21.056 INFO:tasks.workunit.client.0.smithi016.stdout: "read_tier": -1, 2020-03-11T22:35:21.056 INFO:tasks.workunit.client.0.smithi016.stdout: "write_tier": -1, 2020-03-11T22:35:21.057 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_mode": "none", 2020-03-11T22:35:21.057 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_bytes": 0, 2020-03-11T22:35:21.057 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_objects": 0, 2020-03-11T22:35:21.057 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:35:21.057 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:35:21.057 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:35:21.058 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_flush_age": 0, 2020-03-11T22:35:21.058 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_evict_age": 0, 2020-03-11T22:35:21.058 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profile": "", 2020-03-11T22:35:21.058 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_params": { 2020-03-11T22:35:21.058 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "none" 2020-03-11T22:35:21.058 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.059 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_period": 0, 2020-03-11T22:35:21.059 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_count": 0, 2020-03-11T22:35:21.059 INFO:tasks.workunit.client.0.smithi016.stdout: "use_gmt_hitset": true, 2020-03-11T22:35:21.059 INFO:tasks.workunit.client.0.smithi016.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:35:21.059 INFO:tasks.workunit.client.0.smithi016.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:35:21.059 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:35:21.059 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:35:21.059 INFO:tasks.workunit.client.0.smithi016.stdout: "grade_table": [], 2020-03-11T22:35:21.060 INFO:tasks.workunit.client.0.smithi016.stdout: "stripe_width": 0, 2020-03-11T22:35:21.060 INFO:tasks.workunit.client.0.smithi016.stdout: "expected_num_objects": 0, 2020-03-11T22:35:21.061 INFO:tasks.workunit.client.0.smithi016.stdout: "fast_read": false, 2020-03-11T22:35:21.061 INFO:tasks.workunit.client.0.smithi016.stdout: "options": {}, 2020-03-11T22:35:21.061 INFO:tasks.workunit.client.0.smithi016.stdout: "application_metadata": {} 2020-03-11T22:35:21.061 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.061 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.061 INFO:tasks.workunit.client.0.smithi016.stdout: "osds": [ 2020-03-11T22:35:21.062 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.062 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:35:21.062 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "be112234-e19a-4d23-85f2-fc6f4baa8e51", 2020-03-11T22:35:21.062 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:35:21.062 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:35:21.062 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:35:21.062 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:35:21.063 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 5, 2020-03-11T22:35:21.063 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 28, 2020-03-11T22:35:21.063 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 34, 2020-03-11T22:35:21.063 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 39, 2020-03-11T22:35:21.063 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 29, 2020-03-11T22:35:21.064 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:35:21.064 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:35:21.064 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.064 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.064 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.064 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:35:21.064 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22585 2020-03-11T22:35:21.064 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.065 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.065 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.065 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:35:21.065 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22585 2020-03-11T22:35:21.065 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.065 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.065 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.066 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:35:21.066 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.066 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.066 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.066 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:35:21.066 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22585 2020-03-11T22:35:21.066 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.066 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.067 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.067 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:35:21.067 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22585 2020-03-11T22:35:21.067 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.067 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.067 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.067 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:21.068 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.068 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.068 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.068 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:35:21.068 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22585 2020-03-11T22:35:21.068 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.068 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.069 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.069 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:35:21.069 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22585 2020-03-11T22:35:21.069 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.069 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.069 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.069 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:21.070 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.070 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.070 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.071 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:35:21.071 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22585 2020-03-11T22:35:21.072 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.072 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.072 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.072 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:35:21.072 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22585 2020-03-11T22:35:21.072 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.073 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.073 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.073 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6803/22585", 2020-03-11T22:35:21.073 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6805/22585", 2020-03-11T22:35:21.074 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6809/22585", 2020-03-11T22:35:21.074 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6807/22585", 2020-03-11T22:35:21.074 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:35:21.074 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:35:21.074 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:35:21.074 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.074 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.075 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.075 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:35:21.075 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "27627ef5-251f-4ae1-93b4-c3a2f74ee2f5", 2020-03-11T22:35:21.075 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 0, 2020-03-11T22:35:21.075 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:35:21.075 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:35:21.075 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:35:21.076 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 0, 2020-03-11T22:35:21.076 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 0, 2020-03-11T22:35:21.076 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 10, 2020-03-11T22:35:21.076 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 23, 2020-03-11T22:35:21.076 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 30, 2020-03-11T22:35:21.076 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:35:21.076 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:35:21.077 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.077 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.077 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.077 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:35:21.077 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 18203 2020-03-11T22:35:21.077 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.077 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.078 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.078 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:35:21.078 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 18203 2020-03-11T22:35:21.078 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.078 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.078 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.078 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:35:21.079 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.079 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.079 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.079 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:35:21.079 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 18203 2020-03-11T22:35:21.079 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.079 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.080 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.080 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:35:21.080 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 18203 2020-03-11T22:35:21.080 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.080 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.081 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.081 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:21.081 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.081 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.081 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.081 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:35:21.081 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 18203 2020-03-11T22:35:21.082 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.082 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.082 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.082 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:35:21.082 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 18203 2020-03-11T22:35:21.082 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.082 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.082 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.083 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:21.083 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.083 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.083 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.083 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:35:21.083 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 18203 2020-03-11T22:35:21.084 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.084 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.084 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.084 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:35:21.084 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 18203 2020-03-11T22:35:21.084 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.084 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.084 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.085 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/18203", 2020-03-11T22:35:21.085 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/18203", 2020-03-11T22:35:21.085 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/18203", 2020-03-11T22:35:21.085 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/18203", 2020-03-11T22:35:21.085 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:35:21.085 INFO:tasks.workunit.client.0.smithi016.stdout: "exists" 2020-03-11T22:35:21.085 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.086 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.086 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.086 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:35:21.086 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "058aee94-40ae-4d51-8e51-ca60b41dbcb5", 2020-03-11T22:35:21.086 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:35:21.086 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:35:21.086 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:35:21.087 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:35:21.087 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 15, 2020-03-11T22:35:21.087 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 28, 2020-03-11T22:35:21.087 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 35, 2020-03-11T22:35:21.087 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 0, 2020-03-11T22:35:21.087 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 31, 2020-03-11T22:35:21.087 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:35:21.088 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:35:21.088 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.088 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.088 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.088 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:35:21.088 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22779 2020-03-11T22:35:21.088 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.089 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.089 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.089 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:35:21.089 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22779 2020-03-11T22:35:21.089 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.089 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.089 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.090 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:35:21.090 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.090 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.090 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.090 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:35:21.090 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22779 2020-03-11T22:35:21.090 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.091 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.091 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.091 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:35:21.091 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22779 2020-03-11T22:35:21.091 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.091 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.091 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.092 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:21.092 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.092 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.092 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.092 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:35:21.092 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22779 2020-03-11T22:35:21.092 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.093 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.093 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.093 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:35:21.093 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22779 2020-03-11T22:35:21.093 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.093 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.093 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.093 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:21.094 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:35:21.094 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.094 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:35:21.094 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:35:21.094 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22779 2020-03-11T22:35:21.094 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.095 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.095 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:35:21.095 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:35:21.095 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 22779 2020-03-11T22:35:21.095 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.095 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.095 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.096 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/22779", 2020-03-11T22:35:21.096 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/22779", 2020-03-11T22:35:21.096 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22779", 2020-03-11T22:35:21.096 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22779", 2020-03-11T22:35:21.096 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:35:21.096 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:35:21.096 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:35:21.097 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.097 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.097 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.098 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_xinfo": [ 2020-03-11T22:35:21.098 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.098 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:35:21.098 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:34:37.361464+0000", 2020-03-11T22:35:21.098 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:35:21.098 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:35:21.099 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:35:21.099 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:35:21.099 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:33:45.041242+0000", 2020-03-11T22:35:21.099 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:35:21.099 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.099 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.099 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:35:21.100 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:34:37.859234+0000", 2020-03-11T22:35:21.100 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:35:21.100 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:35:21.100 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:35:21.100 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:35:21.100 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:33:51.582571+0000", 2020-03-11T22:35:21.100 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:35:21.100 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.101 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.101 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:35:21.101 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:34:38.392289+0000", 2020-03-11T22:35:21.101 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:35:21.101 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:35:21.101 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:35:21.102 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:35:21.102 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:33:57.942352+0000", 2020-03-11T22:35:21.102 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:35:21.102 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.102 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.102 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap": [], 2020-03-11T22:35:21.102 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap_items": [], 2020-03-11T22:35:21.103 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_temp": [], 2020-03-11T22:35:21.103 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_temp": [], 2020-03-11T22:35:21.103 INFO:tasks.workunit.client.0.smithi016.stdout: "blacklist": {}, 2020-03-11T22:35:21.103 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profiles": { 2020-03-11T22:35:21.103 INFO:tasks.workunit.client.0.smithi016.stdout: "default": { 2020-03-11T22:35:21.103 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:35:21.103 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "2", 2020-03-11T22:35:21.103 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:35:21.104 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van" 2020-03-11T22:35:21.104 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.104 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.104 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps_queue": [], 2020-03-11T22:35:21.104 INFO:tasks.workunit.client.0.smithi016.stdout: "new_removed_snaps": [], 2020-03-11T22:35:21.104 INFO:tasks.workunit.client.0.smithi016.stdout: "new_purged_snaps": [], 2020-03-11T22:35:21.104 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_node_flags": {}, 2020-03-11T22:35:21.105 INFO:tasks.workunit.client.0.smithi016.stdout: "device_class_flags": {} 2020-03-11T22:35:21.105 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.105 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_metadata": [ 2020-03-11T22:35:21.105 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.105 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:35:21.105 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:35:21.105 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6804/22585,v1:127.0.0.1:6805/22585]", 2020-03-11T22:35:21.106 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:35:21.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:35:21.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:21.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:21.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:21.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:21.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:21.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:21.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:21.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:35:21.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:21.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:21.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:21.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:21.108 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:35:21.108 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:21.108 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:21.108 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:35:21.108 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:35:21.108 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:35:21.109 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:21.109 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:35:21.109 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:35:21.109 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:21.109 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:35:21.109 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6802/22585,v1:127.0.0.1:6803/22585]", 2020-03-11T22:35:21.109 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:35:21.110 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/22585,v1:127.0.0.1:6809/22585]", 2020-03-11T22:35:21.110 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/22585,v1:127.0.0.1:6807/22585]", 2020-03-11T22:35:21.110 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:35:21.110 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:35:21.110 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:21.110 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:21.110 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:21.111 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:35:21.111 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:21.111 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:21.111 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:35:21.111 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:35:21.111 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:21.111 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:35:21.112 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.112 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.112 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:35:21.112 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:35:21.112 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/18203,v1:127.0.0.1:6813/18203]", 2020-03-11T22:35:21.112 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:35:21.112 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:35:21.113 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:21.113 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:21.113 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:21.113 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:21.113 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:21.113 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:21.113 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:21.114 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:35:21.114 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:21.114 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:21.114 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:21.114 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:21.114 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:35:21.114 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:21.115 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:21.115 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:35:21.115 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:35:21.115 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:35:21.115 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:21.115 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:35:21.115 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:35:21.116 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:21.116 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:35:21.116 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/18203,v1:127.0.0.1:6811/18203]", 2020-03-11T22:35:21.116 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:35:21.116 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/18203,v1:127.0.0.1:6817/18203]", 2020-03-11T22:35:21.116 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/18203,v1:127.0.0.1:6815/18203]", 2020-03-11T22:35:21.116 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:35:21.117 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:35:21.117 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:21.117 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:21.117 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:21.117 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:35:21.117 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:21.117 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:21.118 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:35:21.118 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:35:21.118 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:21.118 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:35:21.118 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.118 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.118 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:35:21.119 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:35:21.119 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/22779,v1:127.0.0.1:6813/22779]", 2020-03-11T22:35:21.119 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:35:21.119 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:35:21.119 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:21.119 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:21.120 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:21.120 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:21.120 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:21.120 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:21.120 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:21.120 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:35:21.120 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:21.121 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:21.121 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:21.121 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:21.121 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:35:21.121 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:21.121 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:21.121 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:35:21.122 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:35:21.122 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:35:21.122 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:21.122 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:35:21.122 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:35:21.122 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:21.122 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:35:21.123 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/22779,v1:127.0.0.1:6811/22779]", 2020-03-11T22:35:21.123 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:35:21.123 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22779,v1:127.0.0.1:6817/22779]", 2020-03-11T22:35:21.123 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22779,v1:127.0.0.1:6815/22779]", 2020-03-11T22:35:21.123 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:35:21.123 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:35:21.123 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:21.124 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:21.124 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:21.124 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:35:21.124 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:21.124 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:21.124 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:35:21.124 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:35:21.125 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:21.125 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:35:21.125 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.125 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.125 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_first_committed": 1, 2020-03-11T22:35:21.125 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_last_committed": 40, 2020-03-11T22:35:21.125 INFO:tasks.workunit.client.0.smithi016.stdout: "crushmap": { 2020-03-11T22:35:21.126 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": [ 2020-03-11T22:35:21.126 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.126 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:35:21.126 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.0", 2020-03-11T22:35:21.126 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:35:21.126 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.126 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.127 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:35:21.127 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.1", 2020-03-11T22:35:21.127 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:35:21.127 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.127 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.127 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:35:21.127 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.2", 2020-03-11T22:35:21.127 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:35:21.128 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.128 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.128 INFO:tasks.workunit.client.0.smithi016.stdout: "types": [ 2020-03-11T22:35:21.128 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.128 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 0, 2020-03-11T22:35:21.128 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd" 2020-03-11T22:35:21.128 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.129 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.129 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:35:21.129 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "host" 2020-03-11T22:35:21.129 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.129 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.129 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 2, 2020-03-11T22:35:21.129 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "chassis" 2020-03-11T22:35:21.130 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.130 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.130 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 3, 2020-03-11T22:35:21.130 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "rack" 2020-03-11T22:35:21.130 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.130 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.131 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 4, 2020-03-11T22:35:21.131 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "row" 2020-03-11T22:35:21.131 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.131 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.131 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 5, 2020-03-11T22:35:21.131 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pdu" 2020-03-11T22:35:21.131 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.132 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.132 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 6, 2020-03-11T22:35:21.132 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pod" 2020-03-11T22:35:21.132 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.132 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.132 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 7, 2020-03-11T22:35:21.132 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "room" 2020-03-11T22:35:21.133 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.133 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.133 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 8, 2020-03-11T22:35:21.133 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "datacenter" 2020-03-11T22:35:21.133 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.133 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.133 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 9, 2020-03-11T22:35:21.134 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "zone" 2020-03-11T22:35:21.134 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.134 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.134 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 10, 2020-03-11T22:35:21.134 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "region" 2020-03-11T22:35:21.134 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.134 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:35:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "root" 2020-03-11T22:35:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: "buckets": [ 2020-03-11T22:35:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -1, 2020-03-11T22:35:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default", 2020-03-11T22:35:21.136 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:35:21.136 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:35:21.136 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:35:21.136 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:35:21.136 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:35:21.136 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:35:21.136 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.137 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:35:21.137 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:35:21.137 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:35:21.137 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.137 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.137 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.137 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -2, 2020-03-11T22:35:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default~hdd", 2020-03-11T22:35:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:35:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:35:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:35:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:35:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:35:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:35:21.139 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.139 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:35:21.139 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:35:21.139 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:35:21.139 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.139 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.139 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.140 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.140 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:35:21.140 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016", 2020-03-11T22:35:21.140 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:35:21.140 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:35:21.140 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:35:21.140 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:35:21.141 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:35:21.141 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:35:21.141 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.141 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:35:21.141 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:35:21.141 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:35:21.141 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.142 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.142 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:35:21.142 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:35:21.142 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:35:21.142 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.142 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.142 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:35:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:35:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:35:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:35:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016~hdd", 2020-03-11T22:35:21.144 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:35:21.144 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:35:21.144 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:35:21.144 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:35:21.144 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:35:21.144 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:35:21.144 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.145 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:35:21.145 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:35:21.145 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:35:21.145 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.145 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.145 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:35:21.145 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:35:21.146 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:35:21.146 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.146 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.146 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:35:21.146 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:35:21.146 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:35:21.146 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.147 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.147 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.147 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.147 INFO:tasks.workunit.client.0.smithi016.stdout: "rules": [ 2020-03-11T22:35:21.147 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.147 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 0, 2020-03-11T22:35:21.147 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "replicated_rule", 2020-03-11T22:35:21.148 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 0, 2020-03-11T22:35:21.148 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:35:21.148 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 1, 2020-03-11T22:35:21.148 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 10, 2020-03-11T22:35:21.148 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.149 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_firstn", 2020-03-11T22:35:21.150 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:35:21.150 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:35:21.150 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.150 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.150 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:35:21.150 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.150 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:35:21.150 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.151 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.151 INFO:tasks.workunit.client.0.smithi016.stdout: "tunables": { 2020-03-11T22:35:21.151 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_tries": 0, 2020-03-11T22:35:21.151 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:35:21.151 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_total_tries": 50, 2020-03-11T22:35:21.151 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:35:21.151 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:35:21.152 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_stable": 1, 2020-03-11T22:35:21.152 INFO:tasks.workunit.client.0.smithi016.stdout: "straw_calc_version": 1, 2020-03-11T22:35:21.152 INFO:tasks.workunit.client.0.smithi016.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:35:21.152 INFO:tasks.workunit.client.0.smithi016.stdout: "profile": "jewel", 2020-03-11T22:35:21.152 INFO:tasks.workunit.client.0.smithi016.stdout: "optimal_tunables": 1, 2020-03-11T22:35:21.152 INFO:tasks.workunit.client.0.smithi016.stdout: "legacy_tunables": 0, 2020-03-11T22:35:21.152 INFO:tasks.workunit.client.0.smithi016.stdout: "minimum_required_version": "jewel", 2020-03-11T22:35:21.153 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables": 1, 2020-03-11T22:35:21.153 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables2": 1, 2020-03-11T22:35:21.153 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v2_rules": 0, 2020-03-11T22:35:21.153 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables3": 1, 2020-03-11T22:35:21.153 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v3_rules": 0, 2020-03-11T22:35:21.153 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v4_buckets": 1, 2020-03-11T22:35:21.153 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables5": 1, 2020-03-11T22:35:21.153 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v5_rules": 0 2020-03-11T22:35:21.154 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.154 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_args": {} 2020-03-11T22:35:21.154 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.154 INFO:tasks.workunit.client.0.smithi016.stdout: "fsmap": { 2020-03-11T22:35:21.154 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:35:21.154 INFO:tasks.workunit.client.0.smithi016.stdout: "default_fscid": -1, 2020-03-11T22:35:21.154 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": { 2020-03-11T22:35:21.155 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": {}, 2020-03-11T22:35:21.155 INFO:tasks.workunit.client.0.smithi016.stdout: "ro_compat": {}, 2020-03-11T22:35:21.155 INFO:tasks.workunit.client.0.smithi016.stdout: "incompat": { 2020-03-11T22:35:21.155 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_1": "base v0.20", 2020-03-11T22:35:21.155 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:35:21.155 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:35:21.155 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:35:21.156 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:35:21.156 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:35:21.156 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_8": "no anchor table", 2020-03-11T22:35:21.156 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_9": "file layout v2", 2020-03-11T22:35:21.156 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:35:21.156 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.156 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.157 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_flags": { 2020-03-11T22:35:21.157 INFO:tasks.workunit.client.0.smithi016.stdout: "enable_multiple": false, 2020-03-11T22:35:21.157 INFO:tasks.workunit.client.0.smithi016.stdout: "ever_enabled_multiple": false 2020-03-11T22:35:21.157 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.157 INFO:tasks.workunit.client.0.smithi016.stdout: "standbys": [], 2020-03-11T22:35:21.157 INFO:tasks.workunit.client.0.smithi016.stdout: "filesystems": [] 2020-03-11T22:35:21.157 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.158 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:35:21.158 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:35:21.158 INFO:tasks.workunit.client.0.smithi016.stdout: "auth": { 2020-03-11T22:35:21.158 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:35:21.158 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 5, 2020-03-11T22:35:21.158 INFO:tasks.workunit.client.0.smithi016.stdout: "num_secrets": 3 2020-03-11T22:35:21.158 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.159 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:35:21.159 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_active": 1, 2020-03-11T22:35:21.159 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_unknown": 0, 2020-03-11T22:35:21.159 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osd": 3, 2020-03-11T22:35:21.159 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_sum": { 2020-03-11T22:35:21.159 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:35:21.159 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48300, 2020-03-11T22:35:21.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:35:21.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:35:21.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:35:21.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:35:21.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:35:21.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:35:21.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:35:21.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:35:21.161 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:35:21.161 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:35:21.161 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:35:21.161 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:35:21.161 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 100, 2020-03-11T22:35:21.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 100, 2020-03-11T22:35:21.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:35:21.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:35:21.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:35:21.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:35:21.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:35:21.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:35:21.163 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:35:21.163 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:35:21.163 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:35:21.163 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:35:21.163 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:35:21.163 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:35:21.163 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:35:21.164 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:35:21.164 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:35:21.164 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:35:21.164 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:35:21.164 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:35:21.165 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:35:21.165 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:35:21.165 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:35:21.165 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:35:21.165 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:35:21.165 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:35:21.165 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:35:21.166 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.166 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:35:21.166 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:35:21.166 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:35:21.166 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:35:21.166 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 0, 2020-03-11T22:35:21.166 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 0, 2020-03-11T22:35:21.166 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:35:21.167 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:21.167 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:35:21.167 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:35:21.167 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:35:21.167 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.167 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 16, 2020-03-11T22:35:21.167 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 16, 2020-03-11T22:35:21.168 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 2, 2020-03-11T22:35:21.168 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 2, 2020-03-11T22:35:21.168 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 0 2020-03-11T22:35:21.168 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.168 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum": { 2020-03-11T22:35:21.168 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:35:21.168 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:35:21.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:35:21.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:35:21.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:35:21.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:35:21.169 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:35:21.169 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3179328, 2020-03-11T22:35:21.169 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 33344, 2020-03-11T22:35:21.169 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 14, 2020-03-11T22:35:21.170 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145713, 2020-03-11T22:35:21.170 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311393472, 2020-03-11T22:35:21.170 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:35:21.170 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:35:21.170 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318866915328, 2020-03-11T22:35:21.170 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:35:21.170 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 34144256, 2020-03-11T22:35:21.171 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 406574, 2020-03-11T22:35:21.171 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:35:21.171 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:21.171 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:35:21.171 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 14848, 2020-03-11T22:35:21.171 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221210624 2020-03-11T22:35:21.171 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.171 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:35:21.172 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:35:21.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:35:21.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 1, 2020-03-11T22:35:21.172 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:35:21.172 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:35:21.172 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:35:21.173 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.173 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:35:21.173 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:35:21.173 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:35:21.173 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:35:21.173 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:35:21.173 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.174 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:35:21.174 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:35:21.174 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.174 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum_by_class": { 2020-03-11T22:35:21.174 INFO:tasks.workunit.client.0.smithi016.stdout: "hdd": { 2020-03-11T22:35:21.174 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:35:21.174 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:35:21.175 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:35:21.175 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:35:21.175 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:35:21.175 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:35:21.175 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:35:21.175 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3179328, 2020-03-11T22:35:21.176 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 33344, 2020-03-11T22:35:21.176 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 14, 2020-03-11T22:35:21.176 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145713, 2020-03-11T22:35:21.176 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311393472, 2020-03-11T22:35:21.176 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:35:21.176 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:35:21.176 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318866915328, 2020-03-11T22:35:21.177 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:35:21.177 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 34144256, 2020-03-11T22:35:21.177 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 406574, 2020-03-11T22:35:21.177 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:35:21.177 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:21.177 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:35:21.177 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 14848, 2020-03-11T22:35:21.178 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221210624 2020-03-11T22:35:21.178 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.178 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:35:21.178 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:35:21.178 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:35:21.178 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 1, 2020-03-11T22:35:21.178 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:35:21.179 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:35:21.179 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:35:21.179 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.179 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:35:21.179 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:35:21.179 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:35:21.179 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:35:21.180 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:35:21.180 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.180 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:35:21.180 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:35:21.180 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.180 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.180 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_stats": [ 2020-03-11T22:35:21.181 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.181 INFO:tasks.workunit.client.0.smithi016.stdout: "poolid": 1, 2020-03-11T22:35:21.181 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:35:21.181 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:35:21.181 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48300, 2020-03-11T22:35:21.181 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:35:21.182 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:35:21.182 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:35:21.182 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:35:21.182 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:35:21.182 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:35:21.182 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:35:21.182 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:35:21.183 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:35:21.183 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:35:21.183 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:35:21.183 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:35:21.183 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 100, 2020-03-11T22:35:21.183 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 100, 2020-03-11T22:35:21.183 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:35:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:35:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:35:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:35:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:35:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:35:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:35:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:35:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:35:21.185 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:35:21.185 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:35:21.185 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:35:21.185 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:35:21.185 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:35:21.185 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:35:21.185 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:35:21.186 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:35:21.186 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:35:21.186 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:35:21.186 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:35:21.186 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:35:21.186 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:35:21.186 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:35:21.187 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:35:21.187 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:35:21.187 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.187 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:35:21.187 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:35:21.187 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:35:21.187 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:35:21.188 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 19660800, 2020-03-11T22:35:21.188 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 144900, 2020-03-11T22:35:21.188 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:35:21.188 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:21.188 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:35:21.188 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:35:21.188 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:35:21.189 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.189 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 16, 2020-03-11T22:35:21.189 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 16, 2020-03-11T22:35:21.189 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 2, 2020-03-11T22:35:21.189 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 2, 2020-03-11T22:35:21.189 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 3 2020-03-11T22:35:21.189 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.190 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.190 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_stats": [ 2020-03-11T22:35:21.190 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.190 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:35:21.190 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 146028888074 2020-03-11T22:35:21.190 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.190 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.191 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:35:21.191 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 42949672976 2020-03-11T22:35:21.191 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.191 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.191 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:35:21.191 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 150323855370 2020-03-11T22:35:21.191 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_state": [ 2020-03-11T22:35:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "active+undersized+degraded", 2020-03-11T22:35:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 1 2020-03-11T22:35:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_osd": [ 2020-03-11T22:35:21.193 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.193 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:35:21.193 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:35:21.193 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:35:21.193 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:21.193 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.194 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.194 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:35:21.194 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:35:21.194 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 0, 2020-03-11T22:35:21.194 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:21.194 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.194 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.194 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:35:21.195 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 1, 2020-03-11T22:35:21.195 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:35:21.195 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:21.195 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.195 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.195 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": [ 2020-03-11T22:35:21.195 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:35:21.196 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:35:21.196 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": {} 2020-03-11T22:35:21.196 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.196 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:35:21.196 INFO:tasks.workunit.client.0.smithi016.stdout: "servicemap": { 2020-03-11T22:35:21.196 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:35:21.196 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:33:37.921118+0000", 2020-03-11T22:35:21.197 INFO:tasks.workunit.client.0.smithi016.stdout: "services": {} 2020-03-11T22:35:21.197 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:35:21.197 INFO:tasks.workunit.client.0.smithi016.stdout: "paxos": { 2020-03-11T22:35:21.197 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:35:21.197 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 309, 2020-03-11T22:35:21.197 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pn": 0, 2020-03-11T22:35:21.197 INFO:tasks.workunit.client.0.smithi016.stdout: "accepted_pn": 0 2020-03-11T22:35:21.198 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:35:21.198 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:35:21.199 INFO:tasks.workunit.client.0.smithi016.stderr:report 1621390135 2020-03-11T22:35:21.199 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:35:21.199 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-11T22:35:21.199 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-11T22:35:21.199 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-11T22:35:21.200 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:21.200 INFO:tasks.workunit.client.0.smithi016.stdout:writing non-divergent object existing_4 2020-03-11T22:35:21.401 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:21.401 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:35:01.096960+0000 28'100 40:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:35:21.401 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:35:21.401 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:21.402 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:35:21.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:35:21.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-11T22:35:21.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:21.465 INFO:tasks.workunit.client.0.smithi016.stdout:delay recovery 2020-03-11T22:35:21.805 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:21.805 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:35:01.096960+0000 28'100 40:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:35:21.805 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:35:21.806 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:21.806 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:35:21.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:35:21.816 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-11T22:35:21.816 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:35:21.816 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:35:21.817 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:21.817 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:21.817 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:35:21.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.0.asok 2020-03-11T22:35:21.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:21.818 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:35:21.912 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:21.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:35:21.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-11T22:35:21.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:35:21.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:35:21.921 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:21.921 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:21.921 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:35:21.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.2.asok 2020-03-11T22:35:21.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:21.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:35:22.016 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:22.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-11T22:35:22.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:22.025 INFO:tasks.workunit.client.0.smithi016.stdout:revive divergent 1 2020-03-11T22:35:22.365 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:22.365 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:35:01.096960+0000 28'100 40:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:35:22.365 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:35:22.365 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:22.366 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:35:22.374 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-11T22:35:23.007 INFO:tasks.workunit.client.0.smithi016.stderr:noup is set 2020-03-11T22:35:23.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:35:23.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:35:23.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:35:23.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:35:23.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:35:23.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:23.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:23.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:23.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:23.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:23.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:23.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:23.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:23.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:35:23.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:23.020 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:35:23.020 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:23.020 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:23.021 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:23.021 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:23.021 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:35:23.021 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:35:23.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:35:23.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:23.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:23.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:23.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:23.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:23.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:23.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:35:23.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:35:23.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:23.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:35:23.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:23.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:35:23.023 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:35:23.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:35:23.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:35:23.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:35:23.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:35:23.045 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:23.042+0000 7fdd6a8ccec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:23.063 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:23.060+0000 7fdd6a8ccec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:23.072 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:23.062+0000 7fdd6a8ccec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:23.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:35:23.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-11T22:35:23.582 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:23.578+0000 7fdd6a8ccec0 -1 Falling back to public interface 2020-03-11T22:35:24.481 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:24.478+0000 7fdd6a8ccec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:35:28.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-11T22:35:28.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:28.400 INFO:tasks.workunit.client.0.smithi016.stdout:delay recovery divergent 2020-03-11T22:35:28.772 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:28.772 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:35:01.096960+0000 40'101 41:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:35:28.773 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:35:28.773 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:28.774 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:35:28.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-11T22:35:28.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:35:28.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:35:28.783 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:28.783 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:28.783 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:35:28.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.1.asok 2020-03-11T22:35:28.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:28.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:35:28.879 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:28.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-11T22:35:29.815 INFO:tasks.workunit.client.0.smithi016.stderr:noup is unset 2020-03-11T22:35:29.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-11T22:35:29.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:29.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:35:29.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:29.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:29.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:29.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:29.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:29.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:29.825 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:35:30.188 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 34 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/22585,v1:127.0.0.1:6803/22585] [v2:127.0.0.1:6804/22585,v1:127.0.0.1:6805/22585] exists,up be112234-e19a-4d23-85f2-fc6f4baa8e51 2020-03-11T22:35:30.188 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:35:30.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:30.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:30.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:30.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-11T22:35:30.189 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:30.189 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:30.189 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:30.189 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:30.189 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:30.189 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:30.190 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:30.190 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:30.573 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/24247,v1:127.0.0.1:6819/24247] [v2:127.0.0.1:6820/24247,v1:127.0.0.1:6821/24247] exists,up 27627ef5-251f-4ae1-93b4-c3a2f74ee2f5 2020-03-11T22:35:30.573 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:35:30.573 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:30.573 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:30.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:30.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-11T22:35:30.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:30.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:35:30.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:30.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:30.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:30.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:30.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:30.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:30.937 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/22779,v1:127.0.0.1:6811/22779] [v2:127.0.0.1:6812/22779,v1:127.0.0.1:6813/22779] exists,up 058aee94-40ae-4d51-8e51-ca60b41dbcb5 2020-03-11T22:35:31.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:31.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:31.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:31.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:31.276 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:31.276 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:35:01.096960+0000 40'101 41:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:35:31.276 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:35:31.276 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:31.277 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi016.stdout:wait for peering 2020-03-11T22:35:31.634 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:31.634 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:35:01.096960+0000 40'101 41:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:35:31.635 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:35:31.635 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:31.636 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:35:31.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-11T22:35:41.695 INFO:tasks.workunit.client.0.smithi016.stdout:killing divergent 1 2020-03-11T22:35:41.696 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-11T22:35:41.696 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:42.046 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:42.047 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:35:29.952484+0000 40'101 45:132 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:35:42.047 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:35:42.047 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:42.048 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:35:42.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:35:42.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:42.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:42.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:42.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:42.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:42.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:42.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-11T22:35:42.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:42.162 INFO:tasks.workunit.client.0.smithi016.stdout:reviving divergent 1 2020-03-11T22:35:42.502 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:42.502 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:35:29.952484+0000 40'101 45:132 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:35:42.502 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:35:42.502 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:42.503 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:35:42.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:35:42.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:35:42.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:35:42.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:35:42.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:35:42.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:42.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:42.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:42.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:42.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:42.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:42.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:42.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:42.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:35:42.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:42.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:35:42.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:42.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:42.517 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:42.517 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:42.517 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:35:42.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:35:42.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:35:42.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:42.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:42.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:42.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:42.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:42.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:42.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:35:42.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:35:42.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:42.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:35:42.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:42.519 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:35:42.519 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:35:42.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:35:42.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:35:42.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:35:42.521 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:35:42.542 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:42.540+0000 7ff851052ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:42.544 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:42.542+0000 7ff851052ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:42.545 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:42.543+0000 7ff851052ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:42.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:35:42.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:42.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:42.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:42.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:42.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:42.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:42.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:42.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:42.880 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:35:43.064 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:43.061+0000 7ff851052ec0 -1 Falling back to public interface 2020-03-11T22:35:43.271 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/24247,v1:127.0.0.1:6819/24247] [v2:127.0.0.1:6820/24247,v1:127.0.0.1:6821/24247] exists,up 27627ef5-251f-4ae1-93b4-c3a2f74ee2f5 2020-03-11T22:35:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-11T22:35:44.037 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:35:44.035+0000 7ff851052ec0 -1 osd.1 45 log_to_monitors {default=true} 2020-03-11T22:36:03.272 INFO:tasks.workunit.client.0.smithi016.stdout:allowing recovery 2020-03-11T22:36:03.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-11T22:36:03.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:03.627 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:03.627 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:35:45.025952+0000 45'102 48:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:36:03.627 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:36:03.627 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:03.628 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:36:03.637 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:36:03.637 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:36:03.764 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:36:03.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:36:03.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:36:03.899 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:36:03.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:36:03.907 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:36:04.034 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:36:04.042 INFO:tasks.workunit.client.0.smithi016.stdout:reading divergent objects 2020-03-11T22:36:04.042 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-11T22:36:04.042 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:04.381 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:04.381 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:35:45.025952+0000 45'102 48:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:00.277332+0000 0'0 2020-03-11T22:34:00.277332+0000 0 2020-03-11T22:36:04.381 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:36:04.381 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:04.382 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:36:04.391 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-11T22:36:04.392 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-11T22:36:04.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:36:04.393 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:36:04.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:36:04.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:36:04.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:36:04.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:36:04.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:36:04.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:36:04.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-11T22:36:04.520 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:36:04.521 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:36:04.525 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:34:26.003454+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:36:04.525 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:34:26.003454+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:36:04.525 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:36:04.525 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:36:04.525 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:36:04.525 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:36:04.526 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2020-03-11T22:36:04.526 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:34:26.003454+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:36:04.526 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:29.810+0000 7fdd48a87700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:36:04.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:36:04.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:36:04.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-11T22:36:04.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-11T22:36:04.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:36:04.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:36:04.530 INFO:tasks.workunit.client.0.smithi016.stdout:success 2020-03-11T22:36:05.029 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' does not exist 2020-03-11T22:36:05.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-11T22:36:05.038 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:05.038 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:05.038 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:05.038 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:05.038 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:05.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:05.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:36:05.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:36:05.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:36:05.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:36:05.147 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:05.147 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:05.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:05.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:05.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:05.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:05.151 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:36:05.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:36:05.152 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:36:05.153 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:36:05.153 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:36:05.153 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:36:05.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:05.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:36:05.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:36:05.156 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:05.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:36:05.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:36:05.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:36:05.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:36:05.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:05.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:05.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:36:05.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:36:05.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:36:05.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:36:05.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:36:05.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:36:05.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:36:05.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:36:05.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:36:05.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:36:05.171 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:05.171 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:05.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:05.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:05.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:05.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:05.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:36:05.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:36:05.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:36:05.174 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:36:05.174 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:36:05.174 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:36:05.175 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:05.175 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:36:05.176 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:36:05.176 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:05.176 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:36:05.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:36:05.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:36:05.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:36:05.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:05.179 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:05.179 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:36:05.179 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:36:05.179 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:36:05.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:36:05.180 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:36:05.181 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:05.181 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:05.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17411 2020-03-11T22:36:05.182 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:36:05.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:36:05.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:36:05.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:36:05.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-11T22:36:05.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-11T22:36:05.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-11T22:36:05.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-11T22:36:05.184 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-11T22:36:05.184 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-11T22:36:05.185 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-11T22:36:05.185 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-11T22:36:05.185 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:36:05.185 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:36:05.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-11T22:36:05.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:36:05.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:36:05.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:36:05.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:36:05.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:36:05.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:36:05.487 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:36:05.487 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:05.487 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:05.488 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:36:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:36:05.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:36:05.640 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:36:05.640 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:36:05.641 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:36:05.641 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:36:05.641 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:36:05.641 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:36:05.641 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:36:05.641 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:36:05.641 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:05.642 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:05.642 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:05.642 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-mon.a.asok 2020-03-11T22:36:05.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:36:05.643 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17411/ceph-mon.a.asok config get fsid 2020-03-11T22:36:05.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:36:05.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:36:05.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:36:05.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:36:05.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:36:05.736 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:36:05.736 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:36:05.736 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:36:05.736 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:05.736 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:05.736 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:05.736 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-mon.a.asok 2020-03-11T22:36:05.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:36:05.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17411/ceph-mon.a.asok config get mon_host 2020-03-11T22:36:05.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-11T22:36:05.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:36:05.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:36:05.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:36:05.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:36:05.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:36:05.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:36:06.723 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:36:06.723 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:06.724 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:06.724 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:06.724 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:06.724 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:06.724 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:36:06.724 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:36:06.725 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:36:06.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:36:06.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-11T22:36:06.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:06.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:06.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:36:06.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:06.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:36:06.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:06.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:06.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:06.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:06.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:36:06.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:36:06.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:06.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:06.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:06.755 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:06.755 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:06.755 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:06.756 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:06.756 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:06.756 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:06.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:36:06.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:36:06.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:06.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:06.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:06.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:06.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:06.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:06.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:06.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:06.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:06.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:06.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:36:06.758 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:06.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=af406440-1a7b-405f-b1f1-ab2ed8bfbc14 2020-03-11T22:36:06.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 af406440-1a7b-405f-b1f1-ab2ed8bfbc14' 2020-03-11T22:36:06.759 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:06.759 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 af406440-1a7b-405f-b1f1-ab2ed8bfbc14 2020-03-11T22:36:06.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDWZ2le3vLgLRAA3Sg/aqVDZR87yTaKtClzog== 2020-03-11T22:36:06.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDWZ2le3vLgLRAA3Sg/aqVDZR87yTaKtClzog=="}' 2020-03-11T22:36:06.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new af406440-1a7b-405f-b1f1-ab2ed8bfbc14 -i td/divergent-priors/0/new.json 2020-03-11T22:36:07.105 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:36:07.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:36:07.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDWZ2le3vLgLRAA3Sg/aqVDZR87yTaKtClzog== --osd-uuid af406440-1a7b-405f-b1f1-ab2ed8bfbc14 2020-03-11T22:36:07.145 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:07.141+0000 7f305ac8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:07.146 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:07.143+0000 7f305ac8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:07.149 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:07.145+0000 7f305ac8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:07.149 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:07.145+0000 7f305ac8dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:07.149 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:07.145+0000 7f305ac8dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:07.149 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:07.145+0000 7f305ac8dec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:07.187 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:07.184+0000 7f305ac8dec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:36:10.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:36:10.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:10.279 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:36:10.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:36:10.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:10.762 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:36:10.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:36:10.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:10.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:10.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:10.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:10.790 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:10.787+0000 7ff13baddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:10.799 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:10.795+0000 7ff13baddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:10.801 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:10.796+0000 7ff13baddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:11.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:36:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:36:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:11.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:11.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:11.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:11.133 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:36:11.317 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:11.313+0000 7ff13baddec0 -1 Falling back to public interface 2020-03-11T22:36:11.550 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:12.015 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:12.011+0000 7ff13baddec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:36:12.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:12.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:12.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:12.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:12.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:12.553 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:36:12.972 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25549,v1:127.0.0.1:6803/25549] [v2:127.0.0.1:6804/25549,v1:127.0.0.1:6805/25549] exists,up af406440-1a7b-405f-b1f1-ab2ed8bfbc14 2020-03-11T22:36:12.972 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 f4026fd4-5a11-42e2-b606-f350d010eed0 2020-03-11T22:36:12.972 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:36:12.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:36:12.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:12.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:12.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:12.975 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:12.975 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:12.975 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:12.976 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:12.976 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:12.976 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:12.976 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:36:12.976 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:36:12.976 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:12.976 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:12.977 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:12.977 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:12.977 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:12.977 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:12.977 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:12.977 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:12.977 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:12.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:12.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:36:12.978 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:12.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f4026fd4-5a11-42e2-b606-f350d010eed0 2020-03-11T22:36:12.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 f4026fd4-5a11-42e2-b606-f350d010eed0' 2020-03-11T22:36:12.978 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:12.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDcZ2legcPoOhAA/w6hKx/4ERc3en0jkL202Q== 2020-03-11T22:36:12.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDcZ2legcPoOhAA/w6hKx/4ERc3en0jkL202Q=="}' 2020-03-11T22:36:12.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f4026fd4-5a11-42e2-b606-f350d010eed0 -i td/divergent-priors/1/new.json 2020-03-11T22:36:13.472 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:36:13.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:36:13.483 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDcZ2legcPoOhAA/w6hKx/4ERc3en0jkL202Q== --osd-uuid f4026fd4-5a11-42e2-b606-f350d010eed0 2020-03-11T22:36:13.512 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:13.509+0000 7f6deffceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:13.530 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:13.526+0000 7f6deffceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:13.532 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:13.528+0000 7f6deffceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:13.532 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:13.529+0000 7f6deffceec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:13.532 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:13.529+0000 7f6deffceec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:13.533 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:13.529+0000 7f6deffceec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:13.588 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:13.585+0000 7f6deffceec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:36:16.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:36:16.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:16.605 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:36:16.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:36:16.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:17.093 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:36:17.093 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:17.093 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:17.093 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:17.093 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:17.094 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:36:17.120 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:17.117+0000 7f0368189ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:17.122 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:17.119+0000 7f0368189ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:17.123 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:17.120+0000 7f0368189ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:17.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:36:17.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:17.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:36:17.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:17.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:17.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:17.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:17.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:17.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:36:17.559 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:36:17.642 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:17.638+0000 7f0368189ec0 -1 Falling back to public interface 2020-03-11T22:36:17.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:18.444 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:18.441+0000 7f0368189ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:36:18.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:18.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:18.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:18.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:18.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:36:18.992 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:36:19.448 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25904,v1:127.0.0.1:6811/25904] [v2:127.0.0.1:6812/25904,v1:127.0.0.1:6813/25904] exists,up f4026fd4-5a11-42e2-b606-f350d010eed0 2020-03-11T22:36:19.448 INFO:tasks.workunit.client.0.smithi016.stdout:add osd2 8b8190ff-543f-4f23-9247-1ee5e94a2b36 2020-03-11T22:36:19.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:19.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:19.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:19.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:36:19.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-11T22:36:19.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:19.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:19.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:36:19.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:19.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:36:19.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:19.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:19.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:19.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:19.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:36:19.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:36:19.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:19.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:19.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:19.451 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:19.451 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:19.451 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:19.452 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:19.452 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:19.452 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:19.452 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:36:19.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:36:19.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:19.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:19.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:19.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:19.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:19.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:19.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:19.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:19.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:19.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:19.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:36:19.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:19.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8b8190ff-543f-4f23-9247-1ee5e94a2b36 2020-03-11T22:36:19.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 8b8190ff-543f-4f23-9247-1ee5e94a2b36' 2020-03-11T22:36:19.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:19.466 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDjZ2le/ROqGxAA7Jdx4HLqOXYjQ5Erfv01zQ== 2020-03-11T22:36:19.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDjZ2le/ROqGxAA7Jdx4HLqOXYjQ5Erfv01zQ=="}' 2020-03-11T22:36:19.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8b8190ff-543f-4f23-9247-1ee5e94a2b36 -i td/divergent-priors/2/new.json 2020-03-11T22:36:20.173 INFO:tasks.workunit.client.0.smithi016.stdout:2 2020-03-11T22:36:20.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:36:20.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDjZ2le/ROqGxAA7Jdx4HLqOXYjQ5Erfv01zQ== --osd-uuid 8b8190ff-543f-4f23-9247-1ee5e94a2b36 2020-03-11T22:36:20.213 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:20.210+0000 7f121ce6fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:20.236 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:20.233+0000 7f121ce6fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:20.238 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:20.234+0000 7f121ce6fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:20.238 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:20.235+0000 7f121ce6fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:20.238 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:20.235+0000 7f121ce6fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:20.239 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:20.235+0000 7f121ce6fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:20.378 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:20.375+0000 7f121ce6fec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:36:23.479 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:36:23.479 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:23.480 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:36:23.480 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:23.480 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd2 key to auth repository 2020-03-11T22:36:23.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:36:23.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:23.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:23.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:23.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:23.966 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:36:23.993 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:23.990+0000 7ffb9cef7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:23.995 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:23.991+0000 7ffb9cef7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:23.996 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:23.993+0000 7ffb9cef7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:24.336 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:36:24.336 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:24.336 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:36:24.336 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:24.336 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:24.336 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:24.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:24.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:24.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:24.337 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:36:24.514 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:24.510+0000 7ffb9cef7ec0 -1 Falling back to public interface 2020-03-11T22:36:24.740 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:25.278 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:36:25.275+0000 7ffb9cef7ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:36:25.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:25.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:25.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:25.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:25.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:25.743 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:36:26.118 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26259,v1:127.0.0.1:6819/26259] [v2:127.0.0.1:6820/26259,v1:127.0.0.1:6821/26259] exists,up 8b8190ff-543f-4f23-9247-1ee5e94a2b36 2020-03-11T22:36:26.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:26.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:26.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:26.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-11T22:36:26.823 INFO:tasks.workunit.client.0.smithi016.stderr:noout is set 2020-03-11T22:36:26.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-11T22:36:27.525 INFO:tasks.workunit.client.0.smithi016.stderr:noin is set 2020-03-11T22:36:27.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-11T22:36:28.320 INFO:tasks.workunit.client.0.smithi016.stderr:nodown is set 2020-03-11T22:36:28.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-11T22:36:28.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:36:28.939 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' already exists 2020-03-11T22:36:28.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:36:29.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-11T22:36:30.664 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 size to 3 2020-03-11T22:36:30.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-11T22:36:31.400 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 min_size to 2 2020-03-11T22:36:31.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-11T22:36:31.507 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:31.507 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:31.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:31.773 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:36:31.773 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:36:31.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:31.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:31.774 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:31.940 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:36:31.941 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:36:31.941 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:31.941 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:32.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:36:32.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:36:32.077 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:32.077 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:32.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:36:32.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:36:32.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:32.213 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:36:32.213 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:32.214 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:32.214 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:36:32.214 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:32.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:36:32.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:36:32.215 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:32.215 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:36:32.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:36:32.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:32.584 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:36:32.584 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:32.584 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:32.585 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:36:32.585 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:32.586 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:36:32.586 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:36:32.586 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:32.586 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:36:32.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:36:32.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:32.948 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:36:32.948 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:32.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:32.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:36:32.950 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:32.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:36:32.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:36:32.951 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:32.951 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:36:33.315 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509442 -lt 64424509444 2020-03-11T22:36:33.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:34.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:34.317 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:34.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:36:34.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-11T22:36:34.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:34.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:34.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:34.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:34.695 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:36:34.695 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:34.695 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:34.695 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:34.696 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:34.696 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:34.789 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:34.790 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:34.790 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:34.790 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:34.790 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:35.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:35.154 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:36:35.154 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:36:35.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:35.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:35.155 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:35.310 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:36:35.310 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:36:35.311 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:35.311 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:35.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:36:35.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:36:35.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:35.453 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:35.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:36:35.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-03-11T22:36:35.590 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:35.590 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:36:35.590 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:35.590 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:35.591 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:36:35.591 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:35.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:36:35.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:36:35.592 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:35.592 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:36:35.969 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:36:35.969 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:36.970 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:36.970 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:37.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:36:37.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:37.345 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:36:37.345 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:37.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:37.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:36:37.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:37.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:36:37.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:36:37.348 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:37.348 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:36:37.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:36:37.709 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:37.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:36:37.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:37.710 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:37.710 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:36:37.710 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:37.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:36:37.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:36:37.711 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:37.712 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:36:38.102 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509446 -lt 64424509446 2020-03-11T22:36:38.102 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:38.103 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:38.103 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:38.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:38.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:38.579 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:38.579 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:38.579 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:38.579 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:38.579 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:38.579 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:38.935 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:36:38.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:38.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:38.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:39.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:36:39.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:36:39.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:36:39.406 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-11T22:36:39.406 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-11T22:36:39.749 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:36:39.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-11T22:36:39.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-11T22:36:39.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:39.758 INFO:tasks.workunit.client.0.smithi016.stdout:primary and soon to be divergent is 1 2020-03-11T22:36:40.096 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:40.096 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:36:29.401714+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:36:40.097 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:36:40.097 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:40.098 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:36:40.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-11T22:36:40.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:40.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-11T22:36:40.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-11T22:36:40.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:40.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-11T22:36:40.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-11T22:36:40.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:40.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-11T22:36:40.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-11T22:36:40.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-11T22:36:40.108 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-11T22:36:40.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-11T22:36:40.109 INFO:tasks.workunit.client.0.smithi016.stdout:writing initial objects 2020-03-11T22:36:40.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-11T22:36:40.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-11T22:36:40.330 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.330 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-11T22:36:40.397 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.397 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-11T22:36:40.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-11T22:36:40.570 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.570 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-11T22:36:40.664 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.664 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-11T22:36:40.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-11T22:36:40.813 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.813 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-11T22:36:40.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-11T22:36:40.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:40.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-11T22:36:41.012 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.012 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-11T22:36:41.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-11T22:36:41.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-11T22:36:41.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-11T22:36:41.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-11T22:36:41.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-11T22:36:41.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-11T22:36:41.507 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.508 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-11T22:36:41.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-11T22:36:41.680 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.680 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-11T22:36:41.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-11T22:36:41.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-11T22:36:41.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-11T22:36:41.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-11T22:36:41.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:41.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-11T22:36:42.013 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.014 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-11T22:36:42.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-11T22:36:42.149 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-11T22:36:42.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-11T22:36:42.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-11T22:36:42.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-11T22:36:42.597 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.597 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-11T22:36:42.646 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-11T22:36:42.697 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.697 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-11T22:36:42.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-11T22:36:42.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-11T22:36:42.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-11T22:36:42.930 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.930 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-11T22:36:42.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:42.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-11T22:36:43.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-11T22:36:43.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-11T22:36:43.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-11T22:36:43.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-11T22:36:43.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-11T22:36:43.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-11T22:36:43.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-11T22:36:43.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-11T22:36:43.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-11T22:36:43.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-11T22:36:43.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-11T22:36:43.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-11T22:36:43.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.928 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-11T22:36:43.980 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:43.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-11T22:36:44.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-11T22:36:44.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-11T22:36:44.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-11T22:36:44.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-11T22:36:44.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-11T22:36:44.372 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.373 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-11T22:36:44.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-11T22:36:44.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-11T22:36:44.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-11T22:36:44.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-11T22:36:44.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.721 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-11T22:36:44.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:44.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-11T22:36:45.013 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.013 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-11T22:36:45.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.073 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-11T22:36:45.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-11T22:36:45.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-11T22:36:45.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-11T22:36:45.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.348 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-11T22:36:45.397 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.397 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-11T22:36:45.447 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-11T22:36:45.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-11T22:36:45.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-11T22:36:45.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-11T22:36:45.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-11T22:36:45.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-11T22:36:45.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-11T22:36:46.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-11T22:36:46.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-11T22:36:46.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-11T22:36:46.214 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.214 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-11T22:36:46.348 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.349 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-11T22:36:46.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-11T22:36:46.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-11T22:36:46.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.633 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-11T22:36:46.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-11T22:36:46.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-11T22:36:46.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-11T22:36:46.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-11T22:36:47.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-11T22:36:47.090 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.090 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-11T22:36:47.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-11T22:36:47.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.207 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-11T22:36:47.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-11T22:36:47.305 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.306 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-11T22:36:47.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-11T22:36:47.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:36:47.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-11T22:36:47.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:47.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:47.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:47.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:47.476 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:36:47.476 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:47.476 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:47.476 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:47.477 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:47.477 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:47.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:47.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:47.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:47.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:47.534 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:47.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:47.898 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:36:47.898 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:36:47.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:47.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:47.898 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:48.067 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:48.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:36:48.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:36:48.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:48.202 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:48.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:36:48.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:36:48.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:48.338 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:36:48.338 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:48.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:48.339 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:36:48.340 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:48.340 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:36:48.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:36:48.341 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:48.341 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:36:48.703 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:36:48.703 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:48.704 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:36:48.704 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:48.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:48.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:36:48.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:48.706 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:36:48.706 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:36:48.707 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:48.707 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:36:49.100 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672972 2020-03-11T22:36:49.101 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:50.101 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:50.102 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:50.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672972 2020-03-11T22:36:50.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:50.501 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:36:50.501 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:50.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:50.502 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:50.502 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:36:50.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:36:50.504 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:36:50.504 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:50.504 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:36:50.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:36:50.893 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:50.893 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:50.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:51.438 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:51.438 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:51.438 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:51.438 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:51.439 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:51.439 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:51.439 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:51.439 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:51.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:36:51.788 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:51.789 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:51.789 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:52.262 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:36:52.262 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:36:52.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:36:52.263 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-11T22:36:52.263 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:36:52.263 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:36:52.263 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:36:52.264 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:36:52.664 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-11T22:36:52.665 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-11T22:36:52.665 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:52.665 INFO:tasks.workunit.client.0.smithi016.stdout:blackholing osds 0 2 2020-03-11T22:36:53.006 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:53.006 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:36:29.401714+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:36:53.006 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:36:53.006 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:53.008 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:36:53.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:53.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:36:53.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:36:53.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:36:53.017 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:53.017 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:53.018 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:53.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.0.asok 2020-03-11T22:36:53.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:36:53.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:36:53.108 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:36:53.108 INFO:tasks.workunit.client.0.smithi016.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:36:53.108 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:36:53.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:53.116 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:36:53.116 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:36:53.116 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:36:53.116 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:53.117 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:53.117 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:36:53.117 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.2.asok 2020-03-11T22:36:53.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:36:53.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:36:53.207 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:36:53.207 INFO:tasks.workunit.client.0.smithi016.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:36:53.207 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:36:53.216 INFO:tasks.workunit.client.0.smithi016.stdout:create new divergent objects 2020-03-11T22:36:53.216 INFO:tasks.workunit.client.0.smithi016.stdout:writing divergent objects 2020-03-11T22:36:53.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-11T22:36:53.216 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-11T22:36:53.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:36:53.217 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:36:53.217 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-11T22:36:53.217 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-11T22:36:53.217 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-11T22:36:53.217 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-11T22:36:53.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:36:53.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:36:53.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:36:53.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-11T22:36:53.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:36:53.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-11T22:36:53.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:36:53.219 INFO:tasks.workunit.client.0.smithi016.stdout:remove divergent objects 2020-03-11T22:36:53.220 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-11T22:36:53.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-11T22:36:53.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:36:53.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-11T22:36:53.221 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-11T22:36:53.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-11T22:36:53.222 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-11T22:36:53.222 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-11T22:37:03.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-11T22:37:03.228 INFO:tasks.workunit.client.0.smithi016.stderr:rados: no process found 2020-03-11T22:37:03.229 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-11T22:37:03.229 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:03.229 INFO:tasks.workunit.client.0.smithi016.stdout:killing all the osds 2020-03-11T22:37:03.577 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:03.577 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:36:29.401714+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:37:03.577 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:37:03.578 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:03.579 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:37:03.586 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:37:03.587 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:03.587 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:03.587 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:03.587 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:03.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:03.693 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:03.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:37:03.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-11T22:37:04.229 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already down. 2020-03-11T22:37:04.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:37:04.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-11T22:37:04.754 INFO:tasks.workunit.client.0.smithi016.stderr:osd.1 is already down. 2020-03-11T22:37:04.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:37:04.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-11T22:37:05.267 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already down. 2020-03-11T22:37:05.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:05.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-11T22:37:05.778 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already out. 2020-03-11T22:37:05.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:05.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-11T22:37:06.310 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already out. 2020-03-11T22:37:06.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-11T22:37:06.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:06.319 INFO:tasks.workunit.client.0.smithi016.stdout:bringing up non_divergent 0 2 2020-03-11T22:37:06.672 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:06.672 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:36:29.401714+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:37:06.672 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:37:06.672 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:06.673 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:37:06.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:06.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-11T22:37:06.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:06.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:06.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:37:06.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:06.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:37:06.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:06.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:06.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:06.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:06.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:37:06.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:37:06.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:06.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:06.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:06.687 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:06.687 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:06.688 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:06.688 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:06.688 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:06.688 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:37:06.688 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:37:06.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:37:06.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:06.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:06.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:06.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:06.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:06.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:06.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:06.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:06.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:37:06.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:37:06.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:06.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:37:06.690 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:37:06.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:37:06.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:06.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:06.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:06.712 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:06.709+0000 7f6998567ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:06.714 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:06.711+0000 7f6998567ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:06.716 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:06.713+0000 7f6998567ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:07.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:37:07.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:07.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:07.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:07.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:07.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:07.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:07.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:07.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:07.056 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:37:07.232 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:07.228+0000 7f6998567ec0 -1 Falling back to public interface 2020-03-11T22:37:07.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:08.109 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:08.106+0000 7f6998567ec0 -1 osd.0 29 log_to_monitors {default=true} 2020-03-11T22:37:08.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:08.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:08.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:08.441 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:37:08.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:08.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:08.814 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/30308,v1:127.0.0.1:6803/30308] [v2:127.0.0.1:6804/30308,v1:127.0.0.1:6805/30308] exists,up af406440-1a7b-405f-b1f1-ab2ed8bfbc14 2020-03-11T22:37:08.814 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:37:08.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:08.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:08.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:08.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:08.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-11T22:37:08.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:08.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:08.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:37:08.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:08.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:37:08.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:08.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:08.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:08.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:08.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:37:08.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:37:08.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:08.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:08.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:08.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:08.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:08.818 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:08.818 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:08.818 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:08.818 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:37:08.818 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:37:08.818 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:37:08.818 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:08.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:08.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:08.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:08.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:08.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:08.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:08.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:08.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:37:08.820 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:37:08.820 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:08.820 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:37:08.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:37:08.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:08.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:08.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:08.841 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:08.838+0000 7fe7eb543ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:08.843 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:08.840+0000 7fe7eb543ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:08.845 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:08.842+0000 7fe7eb543ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:09.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:37:09.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:09.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:09.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:09.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:09.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:09.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:09.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:09.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:09.183 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:37:09.365 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:09.362+0000 7fe7eb543ec0 -1 Falling back to public interface 2020-03-11T22:37:09.590 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:10.221 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:37:10.218+0000 7fe7eb543ec0 -1 osd.2 29 log_to_monitors {default=true} 2020-03-11T22:37:10.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:10.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:10.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:10.592 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:37:10.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:10.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:10.967 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/30502,v1:127.0.0.1:6811/30502] [v2:127.0.0.1:6812/30502,v1:127.0.0.1:6813/30502] exists,up 8b8190ff-543f-4f23-9247-1ee5e94a2b36 2020-03-11T22:37:10.968 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:10.968 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:10.968 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:10.968 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:10.968 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-11T22:37:11.545 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already in. 2020-03-11T22:37:11.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:11.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-11T22:37:12.045 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already in. 2020-03-11T22:37:12.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:37:12.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-11T22:37:12.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:12.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:12.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:12.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:12.055 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:37:12.055 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:12.056 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:12.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:12.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:12.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:12.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:12.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:12.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:12.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:12.114 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:12.480 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:12.480 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:37:12.480 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:37:12.480 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:12.481 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:12.481 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:12.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:37:12.617 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363' 2020-03-11T22:37:12.617 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:12.617 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:12.712 INFO:tasks.workunit.client.0.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:37:12.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:37:12.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1-' 2020-03-11T22:37:12.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:12.721 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:12.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:37:12.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1- 2-154618822659' 2020-03-11T22:37:12.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:12.857 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855363 2020-03-11T22:37:12.857 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:12.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:12.858 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855363 2020-03-11T22:37:12.858 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:12.861 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:37:12.861 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2020-03-11T22:37:12.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:12.862 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 150323855363 2020-03-11T22:37:13.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 150323855363 2020-03-11T22:37:13.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:14.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:14.235 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:14.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855363 2020-03-11T22:37:14.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:14.611 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:37:14.611 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:14.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:14.612 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:37:14.612 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:14.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:37:14.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:37:14.615 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:14.615 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 2020-03-11T22:37:15.015 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:37:15.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:37:15.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:15.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822659 2020-03-11T22:37:15.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:15.017 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:15.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822659 2020-03-11T22:37:15.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:15.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:37:15.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2020-03-11T22:37:15.020 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:15.020 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 154618822659 2020-03-11T22:37:15.414 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822659 -lt 154618822659 2020-03-11T22:37:15.415 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:15.415 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:15.415 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:15.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:37:15.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:15.943 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:15.943 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:15.944 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:15.944 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:15.944 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:15.944 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:16.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:16.298 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:16.299 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:16.299 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:16.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:16.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:37:16.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:37:16.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:37:16.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:16.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:37:16.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:16.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:16.876 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:16.876 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:16.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:16.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:16.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:16.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:17.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:17.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:17.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:17.228 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:17.696 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:17.696 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:17.696 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:17.696 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:17.697 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:17.697 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:17.697 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:17.697 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:17.697 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:18.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:18.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:18.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:37:18.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:18.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:37:18.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:18.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:18.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:18.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:18.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:18.368 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:18.368 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:18.368 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:18.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:18.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:18.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:18.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:19.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:19.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:19.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:19.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:19.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:19.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:19.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:19.194 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:19.194 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:19.664 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:19.664 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:19.665 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:37:19.665 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:19.665 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:37:20.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:20.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:20.066 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:20.066 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:20.066 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:20.066 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:20.066 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:20.067 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:20.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:20.422 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:20.422 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:20.423 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:20.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:20.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:20.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:20.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:20.900 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:20.900 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:20.900 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:20.900 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:20.900 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:21.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:21.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:21.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:37:21.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:21.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:37:22.176 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:22.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:22.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:22.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:22.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:22.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:22.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:22.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:22.541 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:22.541 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:22.541 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:22.542 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:23.014 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:23.014 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:23.014 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:23.014 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:23.015 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:23.015 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:23.015 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:23.015 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:23.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:23.483 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:23.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:23.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:37:23.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:23.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:37:25.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:25.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:25.085 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:25.086 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:25.086 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:25.086 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:25.086 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:25.086 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:25.447 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:25.448 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:25.448 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:25.448 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:25.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:25.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:25.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:25.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:25.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:25.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:25.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:25.922 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:25.922 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:26.387 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:26.387 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:26.387 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:37:26.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:26.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:37:29.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:29.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:29.589 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:29.590 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:29.590 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:29.590 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:29.590 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:29.590 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:29.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:29.952 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:29.952 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:29.952 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:30.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:30.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:30.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:30.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:30.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:30.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:30.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:30.463 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:30.463 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:31.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:31.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:31.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:37:31.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:31.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:37:37.460 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:37.460 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:37.461 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:37.462 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:37.462 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:37.462 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:37.462 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:37.462 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:37.823 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:37.824 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:37.824 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:37.824 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:38.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:38.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:38.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:38.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:38.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:38.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:38.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:38.297 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:38.297 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:38.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:38.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:37:46.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:46.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:46.076 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:46.076 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:46.076 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:46.077 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:46.077 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:46.077 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:46.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:46.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:46.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:46.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:46.915 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:46.915 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:46.915 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:46.915 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:46.916 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:46.916 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:46.916 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:46.916 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:46.916 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:47.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:47.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:47.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:37:47.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:37:47.855 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:37:47.856 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_fingerprint": "cb131c3c-fd97-4878-9763-256d0f3d43f9", 2020-03-11T22:37:47.856 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:37:47.856 INFO:tasks.workunit.client.0.smithi016.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:37:47.856 INFO:tasks.workunit.client.0.smithi016.stdout: "timestamp": "2020-03-11T22:37:47.852264+0000", 2020-03-11T22:37:47.856 INFO:tasks.workunit.client.0.smithi016.stdout: "tag": "", 2020-03-11T22:37:47.856 INFO:tasks.workunit.client.0.smithi016.stdout: "health": { 2020-03-11T22:37:47.856 INFO:tasks.workunit.client.0.smithi016.stdout: "status": "HEALTH_WARN", 2020-03-11T22:37:47.857 INFO:tasks.workunit.client.0.smithi016.stdout: "checks": { 2020-03-11T22:37:47.857 INFO:tasks.workunit.client.0.smithi016.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:37:47.857 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:47.857 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:37:47.857 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:37:47.857 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 17 2020-03-11T22:37:47.857 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.858 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [], 2020-03-11T22:37:47.858 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:37:47.858 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.858 INFO:tasks.workunit.client.0.smithi016.stdout: "OSD_DOWN": { 2020-03-11T22:37:47.858 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:47.858 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:37:47.858 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 osds down", 2020-03-11T22:37:47.859 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:37:47.859 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.859 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:37:47.859 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.859 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "osd.1 (root=default,host=smithi016) is down" 2020-03-11T22:37:47.859 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.859 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.860 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:37:47.860 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.860 INFO:tasks.workunit.client.0.smithi016.stdout: "PG_DEGRADED": { 2020-03-11T22:37:47.860 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:47.860 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:37:47.860 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:37:47.860 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:37:47.861 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.861 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:37:47.861 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.861 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:37:47.861 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.861 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.861 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:37:47.862 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.862 INFO:tasks.workunit.client.0.smithi016.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:37:47.862 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:47.862 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:37:47.862 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:37:47.862 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:37:47.862 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.863 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:37:47.863 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.863 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:37:47.863 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.863 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.863 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:37:47.863 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.864 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.864 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:37:47.864 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.864 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.864 INFO:tasks.workunit.client.0.smithi016.stdout: "mutes": [] 2020-03-11T22:37:47.864 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.864 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_first_committed": 1, 2020-03-11T22:37:47.864 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_last_committed": 1, 2020-03-11T22:37:47.865 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap": { 2020-03-11T22:37:47.865 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:37:47.865 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:37:47.865 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:36:05.205097Z", 2020-03-11T22:37:47.865 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:36:05.205097Z", 2020-03-11T22:37:47.865 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release": 15, 2020-03-11T22:37:47.865 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:37:47.866 INFO:tasks.workunit.client.0.smithi016.stdout: "features": { 2020-03-11T22:37:47.866 INFO:tasks.workunit.client.0.smithi016.stdout: "persistent": [ 2020-03-11T22:37:47.866 INFO:tasks.workunit.client.0.smithi016.stdout: "kraken", 2020-03-11T22:37:47.866 INFO:tasks.workunit.client.0.smithi016.stdout: "luminous", 2020-03-11T22:37:47.866 INFO:tasks.workunit.client.0.smithi016.stdout: "mimic", 2020-03-11T22:37:47.866 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap-prune", 2020-03-11T22:37:47.866 INFO:tasks.workunit.client.0.smithi016.stdout: "nautilus", 2020-03-11T22:37:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: "octopus" 2020-03-11T22:37:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: "optional": [] 2020-03-11T22:37:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: "mons": [ 2020-03-11T22:37:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: "rank": 0, 2020-03-11T22:37:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "a", 2020-03-11T22:37:47.868 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:37:47.868 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.868 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.868 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.868 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:37:47.868 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 0 2020-03-11T22:37:47.868 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.869 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.869 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.869 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:37:47.869 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:37:47.869 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 0, 2020-03-11T22:37:47.869 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 0 2020-03-11T22:37:47.869 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.870 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.870 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.870 INFO:tasks.workunit.client.0.smithi016.stdout: "quorum": [ 2020-03-11T22:37:47.870 INFO:tasks.workunit.client.0.smithi016.stdout: 0 2020-03-11T22:37:47.870 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.870 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap": { 2020-03-11T22:37:47.870 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 39, 2020-03-11T22:37:47.870 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:37:47.871 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:36:05.916167+0000", 2020-03-11T22:37:47.871 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:37:12.083146+0000", 2020-03-11T22:37:47.871 INFO:tasks.workunit.client.0.smithi016.stdout: "last_up_change": "2020-03-11T22:37:10.313743+0000", 2020-03-11T22:37:47.871 INFO:tasks.workunit.client.0.smithi016.stdout: "last_in_change": "2020-03-11T22:37:11.863615+0000", 2020-03-11T22:37:47.871 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:37:47.871 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_num": 5800384, 2020-03-11T22:37:47.871 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_set": [ 2020-03-11T22:37:47.872 INFO:tasks.workunit.client.0.smithi016.stdout: "nodown", 2020-03-11T22:37:47.872 INFO:tasks.workunit.client.0.smithi016.stdout: "noin", 2020-03-11T22:37:47.872 INFO:tasks.workunit.client.0.smithi016.stdout: "noout", 2020-03-11T22:37:47.872 INFO:tasks.workunit.client.0.smithi016.stdout: "pglog_hardlimit", 2020-03-11T22:37:47.872 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snapdirs", 2020-03-11T22:37:47.872 INFO:tasks.workunit.client.0.smithi016.stdout: "recovery_deletes", 2020-03-11T22:37:47.872 INFO:tasks.workunit.client.0.smithi016.stdout: "sortbitwise" 2020-03-11T22:37:47.873 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.873 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_version": 7, 2020-03-11T22:37:47.873 INFO:tasks.workunit.client.0.smithi016.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:37:47.873 INFO:tasks.workunit.client.0.smithi016.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:37:47.873 INFO:tasks.workunit.client.0.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:37:47.873 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_snapshot": "", 2020-03-11T22:37:47.873 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_max": 1, 2020-03-11T22:37:47.874 INFO:tasks.workunit.client.0.smithi016.stdout: "max_osd": 3, 2020-03-11T22:37:47.874 INFO:tasks.workunit.client.0.smithi016.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:37:47.874 INFO:tasks.workunit.client.0.smithi016.stdout: "min_compat_client": "jewel", 2020-03-11T22:37:47.874 INFO:tasks.workunit.client.0.smithi016.stdout: "require_osd_release": "octopus", 2020-03-11T22:37:47.874 INFO:tasks.workunit.client.0.smithi016.stdout: "pools": [ 2020-03-11T22:37:47.874 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.874 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:37:47.874 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_name": "test", 2020-03-11T22:37:47.875 INFO:tasks.workunit.client.0.smithi016.stdout: "create_time": "2020-03-11T22:36:28.601035+0000", 2020-03-11T22:37:47.875 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": 1, 2020-03-11T22:37:47.875 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_names": "hashpspool", 2020-03-11T22:37:47.875 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:37:47.875 INFO:tasks.workunit.client.0.smithi016.stdout: "size": 3, 2020-03-11T22:37:47.875 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 2, 2020-03-11T22:37:47.875 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_rule": 0, 2020-03-11T22:37:47.876 INFO:tasks.workunit.client.0.smithi016.stdout: "object_hash": 2, 2020-03-11T22:37:47.876 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:37:47.876 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num": 1, 2020-03-11T22:37:47.876 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num": 1, 2020-03-11T22:37:47.876 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num_target": 1, 2020-03-11T22:37:47.876 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_target": 1, 2020-03-11T22:37:47.876 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_pending": 1, 2020-03-11T22:37:47.877 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pg_merge_meta": { 2020-03-11T22:37:47.877 INFO:tasks.workunit.client.0.smithi016.stdout: "source_pgid": "0.0", 2020-03-11T22:37:47.877 INFO:tasks.workunit.client.0.smithi016.stdout: "ready_epoch": 0, 2020-03-11T22:37:47.877 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_started": 0, 2020-03-11T22:37:47.877 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_clean": 0, 2020-03-11T22:37:47.877 INFO:tasks.workunit.client.0.smithi016.stdout: "source_version": "0'0", 2020-03-11T22:37:47.877 INFO:tasks.workunit.client.0.smithi016.stdout: "target_version": "0'0" 2020-03-11T22:37:47.878 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.878 INFO:tasks.workunit.client.0.smithi016.stdout: "last_change": "29", 2020-03-11T22:37:47.878 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend": "0", 2020-03-11T22:37:47.878 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:37:47.878 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:37:47.878 INFO:tasks.workunit.client.0.smithi016.stdout: "auid": 0, 2020-03-11T22:37:47.878 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:37:47.879 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_seq": 0, 2020-03-11T22:37:47.879 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_epoch": 0, 2020-03-11T22:37:47.879 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_snaps": [], 2020-03-11T22:37:47.879 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps": "[]", 2020-03-11T22:37:47.879 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_bytes": 0, 2020-03-11T22:37:47.879 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_objects": 0, 2020-03-11T22:37:47.879 INFO:tasks.workunit.client.0.smithi016.stdout: "tiers": [], 2020-03-11T22:37:47.880 INFO:tasks.workunit.client.0.smithi016.stdout: "tier_of": -1, 2020-03-11T22:37:47.880 INFO:tasks.workunit.client.0.smithi016.stdout: "read_tier": -1, 2020-03-11T22:37:47.880 INFO:tasks.workunit.client.0.smithi016.stdout: "write_tier": -1, 2020-03-11T22:37:47.880 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_mode": "none", 2020-03-11T22:37:47.880 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_bytes": 0, 2020-03-11T22:37:47.880 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_objects": 0, 2020-03-11T22:37:47.880 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:37:47.880 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:37:47.881 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:37:47.881 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_flush_age": 0, 2020-03-11T22:37:47.881 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_evict_age": 0, 2020-03-11T22:37:47.881 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profile": "", 2020-03-11T22:37:47.881 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_params": { 2020-03-11T22:37:47.881 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "none" 2020-03-11T22:37:47.881 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.882 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_period": 0, 2020-03-11T22:37:47.882 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_count": 0, 2020-03-11T22:37:47.882 INFO:tasks.workunit.client.0.smithi016.stdout: "use_gmt_hitset": true, 2020-03-11T22:37:47.882 INFO:tasks.workunit.client.0.smithi016.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:37:47.882 INFO:tasks.workunit.client.0.smithi016.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:37:47.882 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:37:47.882 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:37:47.883 INFO:tasks.workunit.client.0.smithi016.stdout: "grade_table": [], 2020-03-11T22:37:47.883 INFO:tasks.workunit.client.0.smithi016.stdout: "stripe_width": 0, 2020-03-11T22:37:47.883 INFO:tasks.workunit.client.0.smithi016.stdout: "expected_num_objects": 0, 2020-03-11T22:37:47.883 INFO:tasks.workunit.client.0.smithi016.stdout: "fast_read": false, 2020-03-11T22:37:47.883 INFO:tasks.workunit.client.0.smithi016.stdout: "options": {}, 2020-03-11T22:37:47.883 INFO:tasks.workunit.client.0.smithi016.stdout: "application_metadata": {} 2020-03-11T22:37:47.883 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.883 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.884 INFO:tasks.workunit.client.0.smithi016.stdout: "osds": [ 2020-03-11T22:37:47.884 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.884 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:37:47.884 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "af406440-1a7b-405f-b1f1-ab2ed8bfbc14", 2020-03-11T22:37:47.884 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:37:47.884 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:37:47.884 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:37:47.885 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:37:47.885 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 5, 2020-03-11T22:37:47.885 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 29, 2020-03-11T22:37:47.885 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 35, 2020-03-11T22:37:47.885 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 38, 2020-03-11T22:37:47.885 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 30, 2020-03-11T22:37:47.885 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:37:47.886 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:37:47.886 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.886 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.886 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.886 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:37:47.886 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:37:47.886 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.887 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.887 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.887 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:37:47.887 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:37:47.887 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.887 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.887 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.888 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:37:47.888 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.888 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.888 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.888 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:37:47.888 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:37:47.888 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.889 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.889 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.889 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:37:47.889 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:37:47.889 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.889 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.889 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.889 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:47.890 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.890 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.890 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.890 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:37:47.890 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:37:47.890 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.890 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.891 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.891 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:37:47.891 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:37:47.891 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.891 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.891 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.891 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:47.892 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.892 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.892 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.892 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:37:47.892 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:37:47.892 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.892 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.893 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.893 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:37:47.893 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:37:47.893 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.893 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.893 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.893 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6803/30308", 2020-03-11T22:37:47.894 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6805/30308", 2020-03-11T22:37:47.894 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30308", 2020-03-11T22:37:47.894 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30308", 2020-03-11T22:37:47.894 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:37:47.894 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:37:47.894 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:37:47.894 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.894 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.895 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.895 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:37:47.895 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "f4026fd4-5a11-42e2-b606-f350d010eed0", 2020-03-11T22:37:47.895 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 0, 2020-03-11T22:37:47.895 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:37:47.895 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:37:47.896 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:37:47.896 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 0, 2020-03-11T22:37:47.896 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 0, 2020-03-11T22:37:47.896 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 10, 2020-03-11T22:37:47.896 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 23, 2020-03-11T22:37:47.896 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 31, 2020-03-11T22:37:47.897 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:37:47.897 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:37:47.897 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.897 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.897 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.897 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:37:47.897 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:37:47.898 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.898 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.898 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.898 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:37:47.898 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:37:47.898 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.898 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.899 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.899 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:37:47.899 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.899 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.899 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.899 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:37:47.899 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:37:47.899 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.900 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.900 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.900 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:37:47.900 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:37:47.900 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.900 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.900 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.901 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:47.901 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.901 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.901 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.901 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:37:47.901 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:37:47.901 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.902 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.902 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.902 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:37:47.902 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:37:47.902 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.902 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.902 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.903 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:47.903 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.903 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.903 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.903 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:37:47.903 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:37:47.903 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.903 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.904 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.904 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:37:47.904 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:37:47.904 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.904 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.904 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.904 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/25904", 2020-03-11T22:37:47.905 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/25904", 2020-03-11T22:37:47.905 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25904", 2020-03-11T22:37:47.905 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25904", 2020-03-11T22:37:47.905 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:37:47.905 INFO:tasks.workunit.client.0.smithi016.stdout: "exists" 2020-03-11T22:37:47.905 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.905 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.906 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.906 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:37:47.906 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "8b8190ff-543f-4f23-9247-1ee5e94a2b36", 2020-03-11T22:37:47.906 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:37:47.906 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:37:47.906 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:37:47.906 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:37:47.907 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 15, 2020-03-11T22:37:47.907 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 29, 2020-03-11T22:37:47.907 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 36, 2020-03-11T22:37:47.907 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 0, 2020-03-11T22:37:47.907 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 32, 2020-03-11T22:37:47.907 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:37:47.907 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:37:47.907 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.908 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.908 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.908 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:37:47.908 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:37:47.908 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.908 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.908 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.909 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:37:47.909 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:37:47.909 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.909 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.909 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.909 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:37:47.909 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.910 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.910 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.910 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:37:47.910 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:37:47.910 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.910 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.910 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.911 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:37:47.911 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:37:47.911 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.911 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.911 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.911 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:47.912 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.912 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.912 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.912 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:37:47.912 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:37:47.912 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.912 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.913 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.913 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:37:47.913 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:37:47.913 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.913 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.913 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.913 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:47.914 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:37:47.914 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.914 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:37:47.914 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:37:47.914 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:37:47.914 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.914 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.915 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:37:47.915 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:37:47.915 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:37:47.915 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.915 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.915 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.915 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/30502", 2020-03-11T22:37:47.915 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/30502", 2020-03-11T22:37:47.916 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30502", 2020-03-11T22:37:47.916 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30502", 2020-03-11T22:37:47.916 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:37:47.916 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:37:47.916 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:37:47.916 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.916 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.917 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.917 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_xinfo": [ 2020-03-11T22:37:47.917 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.917 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:37:47.917 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:37:04.006421+0000", 2020-03-11T22:37:47.917 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:37:47.917 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:37:47.918 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:37:47.918 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:37:47.918 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:13.035805+0000", 2020-03-11T22:37:47.918 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:37:47.918 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.918 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.918 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:37:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:37:04.560676+0000", 2020-03-11T22:37:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:37:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:37:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:37:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:37:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:19.439160+0000", 2020-03-11T22:37:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:37:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.920 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.920 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:37:47.920 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:37:05.072866+0000", 2020-03-11T22:37:47.920 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:37:47.920 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:37:47.920 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:37:47.921 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:37:47.921 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:26.266667+0000", 2020-03-11T22:37:47.921 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:37:47.921 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.921 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.921 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap": [], 2020-03-11T22:37:47.921 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap_items": [], 2020-03-11T22:37:47.921 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_temp": [], 2020-03-11T22:37:47.922 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_temp": [], 2020-03-11T22:37:47.922 INFO:tasks.workunit.client.0.smithi016.stdout: "blacklist": {}, 2020-03-11T22:37:47.922 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profiles": { 2020-03-11T22:37:47.922 INFO:tasks.workunit.client.0.smithi016.stdout: "default": { 2020-03-11T22:37:47.922 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:37:47.922 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "2", 2020-03-11T22:37:47.922 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:37:47.923 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van" 2020-03-11T22:37:47.923 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.923 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.923 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps_queue": [], 2020-03-11T22:37:47.923 INFO:tasks.workunit.client.0.smithi016.stdout: "new_removed_snaps": [], 2020-03-11T22:37:47.923 INFO:tasks.workunit.client.0.smithi016.stdout: "new_purged_snaps": [], 2020-03-11T22:37:47.923 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_node_flags": {}, 2020-03-11T22:37:47.924 INFO:tasks.workunit.client.0.smithi016.stdout: "device_class_flags": {} 2020-03-11T22:37:47.924 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.924 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_metadata": [ 2020-03-11T22:37:47.924 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.924 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:37:47.924 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:37:47.924 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6804/30308,v1:127.0.0.1:6805/30308]", 2020-03-11T22:37:47.925 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:37:47.925 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:37:47.925 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:47.925 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:47.925 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:47.925 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:47.925 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:47.926 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:47.926 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:47.926 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:37:47.926 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:47.926 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:47.926 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:47.926 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:47.927 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:37:47.927 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:47.927 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:47.927 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:47.927 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:37:47.927 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:37:47.927 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:47.928 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:37:47.928 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:37:47.928 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:47.928 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:37:47.928 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6802/30308,v1:127.0.0.1:6803/30308]", 2020-03-11T22:37:47.928 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:37:47.928 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30308,v1:127.0.0.1:6809/30308]", 2020-03-11T22:37:47.929 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30308,v1:127.0.0.1:6807/30308]", 2020-03-11T22:37:47.929 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:37:47.929 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:37:47.929 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:47.929 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:47.929 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:47.929 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:37:47.930 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:47.930 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:47.930 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:37:47.930 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:37:47.930 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:47.930 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:37:47.930 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.931 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.931 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:37:47.931 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:37:47.931 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/25904,v1:127.0.0.1:6813/25904]", 2020-03-11T22:37:47.931 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:37:47.931 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:37:47.931 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:47.932 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:47.932 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:47.932 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:47.932 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:47.932 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:47.932 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:47.932 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:37:47.933 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:47.933 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:47.933 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:47.933 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:47.933 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:37:47.933 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:47.933 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:47.933 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:47.934 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:37:47.934 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:37:47.934 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:47.934 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:37:47.934 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:37:47.934 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:47.935 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:37:47.935 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/25904,v1:127.0.0.1:6811/25904]", 2020-03-11T22:37:47.935 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:37:47.935 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25904,v1:127.0.0.1:6817/25904]", 2020-03-11T22:37:47.935 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25904,v1:127.0.0.1:6815/25904]", 2020-03-11T22:37:47.935 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:37:47.935 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:37:47.935 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:47.936 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:47.936 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:47.936 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:37:47.936 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:47.936 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:47.936 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:37:47.936 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:37:47.937 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:47.937 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:37:47.937 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.937 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.937 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:37:47.937 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:37:47.937 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/30502,v1:127.0.0.1:6813/30502]", 2020-03-11T22:37:47.938 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:37:47.938 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:37:47.938 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:47.938 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:47.938 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:47.938 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:47.938 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:47.939 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:47.939 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:47.939 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:37:47.939 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:47.939 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:47.939 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:47.939 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:47.940 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:37:47.940 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:47.940 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:47.940 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:47.940 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:37:47.940 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:37:47.940 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:47.940 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:37:47.941 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:37:47.941 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:47.941 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:37:47.941 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/30502,v1:127.0.0.1:6811/30502]", 2020-03-11T22:37:47.941 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:37:47.941 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30502,v1:127.0.0.1:6817/30502]", 2020-03-11T22:37:47.941 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30502,v1:127.0.0.1:6815/30502]", 2020-03-11T22:37:47.942 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:37:47.942 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:37:47.942 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:47.942 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:47.942 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:47.942 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:37:47.942 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:47.943 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:47.943 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:37:47.943 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:37:47.943 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:47.943 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:37:47.943 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.943 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.943 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_first_committed": 1, 2020-03-11T22:37:47.944 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_last_committed": 39, 2020-03-11T22:37:47.944 INFO:tasks.workunit.client.0.smithi016.stdout: "crushmap": { 2020-03-11T22:37:47.944 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": [ 2020-03-11T22:37:47.944 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.944 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:37:47.944 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.0", 2020-03-11T22:37:47.944 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:37:47.945 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.945 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.945 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:37:47.945 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.1", 2020-03-11T22:37:47.945 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:37:47.945 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.945 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.946 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:37:47.946 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.2", 2020-03-11T22:37:47.946 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:37:47.946 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.946 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.946 INFO:tasks.workunit.client.0.smithi016.stdout: "types": [ 2020-03-11T22:37:47.946 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.946 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 0, 2020-03-11T22:37:47.947 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd" 2020-03-11T22:37:47.947 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.947 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.947 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:37:47.947 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "host" 2020-03-11T22:37:47.947 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.947 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.948 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 2, 2020-03-11T22:37:47.948 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "chassis" 2020-03-11T22:37:47.948 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.948 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.948 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 3, 2020-03-11T22:37:47.948 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "rack" 2020-03-11T22:37:47.948 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.948 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.949 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 4, 2020-03-11T22:37:47.949 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "row" 2020-03-11T22:37:47.949 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.949 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.949 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 5, 2020-03-11T22:37:47.949 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pdu" 2020-03-11T22:37:47.949 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.950 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.950 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 6, 2020-03-11T22:37:47.950 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pod" 2020-03-11T22:37:47.950 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.950 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.950 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 7, 2020-03-11T22:37:47.950 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "room" 2020-03-11T22:37:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 8, 2020-03-11T22:37:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "datacenter" 2020-03-11T22:37:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 9, 2020-03-11T22:37:47.951 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "zone" 2020-03-11T22:37:47.952 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.952 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.952 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 10, 2020-03-11T22:37:47.952 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "region" 2020-03-11T22:37:47.952 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.952 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.952 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:37:47.953 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "root" 2020-03-11T22:37:47.953 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.953 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.953 INFO:tasks.workunit.client.0.smithi016.stdout: "buckets": [ 2020-03-11T22:37:47.953 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.953 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -1, 2020-03-11T22:37:47.953 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default", 2020-03-11T22:37:47.954 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:37:47.954 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:37:47.954 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:37:47.954 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:37:47.954 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:37:47.954 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:37:47.954 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.955 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:37:47.955 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:37:47.955 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:37:47.955 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.955 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.955 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.955 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.956 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -2, 2020-03-11T22:37:47.956 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default~hdd", 2020-03-11T22:37:47.956 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:37:47.956 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:37:47.956 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:37:47.956 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:37:47.956 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:37:47.956 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:37:47.957 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.957 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:37:47.957 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:37:47.957 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:37:47.957 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.957 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.957 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.958 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.958 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:37:47.958 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016", 2020-03-11T22:37:47.958 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:37:47.958 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:37:47.958 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:37:47.958 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:37:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:37:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:37:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:37:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:37:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:37:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.959 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.960 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:37:47.960 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:37:47.960 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:37:47.960 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.960 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.960 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:37:47.960 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:37:47.961 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:37:47.961 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.961 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.961 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.961 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.961 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:37:47.961 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016~hdd", 2020-03-11T22:37:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:37:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:37:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:37:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:37:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:37:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:37:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.962 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:37:47.963 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:37:47.963 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:37:47.963 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.963 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.963 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:37:47.963 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:37:47.963 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:37:47.964 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.964 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.964 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:37:47.964 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:37:47.964 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:37:47.964 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.964 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi016.stdout: "rules": [ 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 0, 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "replicated_rule", 2020-03-11T22:37:47.965 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 0, 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 1, 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 10, 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:37:47.966 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_firstn", 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.967 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.968 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:37:47.968 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.968 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:37:47.968 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.968 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:47.968 INFO:tasks.workunit.client.0.smithi016.stdout: "tunables": { 2020-03-11T22:37:47.968 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_tries": 0, 2020-03-11T22:37:47.969 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:37:47.969 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_total_tries": 50, 2020-03-11T22:37:47.969 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:37:47.969 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:37:47.969 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_stable": 1, 2020-03-11T22:37:47.969 INFO:tasks.workunit.client.0.smithi016.stdout: "straw_calc_version": 1, 2020-03-11T22:37:47.969 INFO:tasks.workunit.client.0.smithi016.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:37:47.970 INFO:tasks.workunit.client.0.smithi016.stdout: "profile": "jewel", 2020-03-11T22:37:47.970 INFO:tasks.workunit.client.0.smithi016.stdout: "optimal_tunables": 1, 2020-03-11T22:37:47.970 INFO:tasks.workunit.client.0.smithi016.stdout: "legacy_tunables": 0, 2020-03-11T22:37:47.970 INFO:tasks.workunit.client.0.smithi016.stdout: "minimum_required_version": "jewel", 2020-03-11T22:37:47.970 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables": 1, 2020-03-11T22:37:47.970 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables2": 1, 2020-03-11T22:37:47.970 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v2_rules": 0, 2020-03-11T22:37:47.970 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables3": 1, 2020-03-11T22:37:47.971 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v3_rules": 0, 2020-03-11T22:37:47.971 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v4_buckets": 1, 2020-03-11T22:37:47.971 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables5": 1, 2020-03-11T22:37:47.971 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v5_rules": 0 2020-03-11T22:37:47.971 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.971 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_args": {} 2020-03-11T22:37:47.971 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.972 INFO:tasks.workunit.client.0.smithi016.stdout: "fsmap": { 2020-03-11T22:37:47.972 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:37:47.972 INFO:tasks.workunit.client.0.smithi016.stdout: "default_fscid": -1, 2020-03-11T22:37:47.972 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": { 2020-03-11T22:37:47.972 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": {}, 2020-03-11T22:37:47.972 INFO:tasks.workunit.client.0.smithi016.stdout: "ro_compat": {}, 2020-03-11T22:37:47.972 INFO:tasks.workunit.client.0.smithi016.stdout: "incompat": { 2020-03-11T22:37:47.973 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_1": "base v0.20", 2020-03-11T22:37:47.973 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:37:47.973 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:37:47.973 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:37:47.973 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:37:47.973 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:37:47.973 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_8": "no anchor table", 2020-03-11T22:37:47.974 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_9": "file layout v2", 2020-03-11T22:37:47.974 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:37:47.974 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.974 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.974 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_flags": { 2020-03-11T22:37:47.974 INFO:tasks.workunit.client.0.smithi016.stdout: "enable_multiple": false, 2020-03-11T22:37:47.974 INFO:tasks.workunit.client.0.smithi016.stdout: "ever_enabled_multiple": false 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi016.stdout: "standbys": [], 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi016.stdout: "filesystems": [] 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi016.stdout: "auth": { 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 5, 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi016.stdout: "num_secrets": 3 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_active": 1, 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_unknown": 0, 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osd": 3, 2020-03-11T22:37:47.977 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_sum": { 2020-03-11T22:37:47.977 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:37:47.977 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48300, 2020-03-11T22:37:47.977 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:37:47.977 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:37:47.977 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:37:47.977 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:37:47.978 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:37:47.978 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:37:47.978 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:37:47.978 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:37:47.978 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:37:47.978 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:37:47.978 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:37:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:37:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 100, 2020-03-11T22:37:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 100, 2020-03-11T22:37:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:37:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:37:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:37:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:37:47.979 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:37:47.980 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:37:47.980 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:37:47.980 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:37:47.980 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:37:47.980 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:37:47.980 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:37:47.980 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:37:47.981 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:37:47.981 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:37:47.981 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:37:47.981 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:37:47.981 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:37:47.981 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:37:47.981 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:37:47.982 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:37:47.982 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:37:47.982 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:37:47.982 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:37:47.982 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:37:47.982 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:37:47.982 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.983 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:37:47.983 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:37:47.983 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:37:47.983 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:37:47.983 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 0, 2020-03-11T22:37:47.983 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 0, 2020-03-11T22:37:47.983 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:37:47.983 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:47.984 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:37:47.984 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:37:47.984 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:37:47.984 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.984 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 16, 2020-03-11T22:37:47.984 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 16, 2020-03-11T22:37:47.984 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 2, 2020-03-11T22:37:47.985 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 2, 2020-03-11T22:37:47.985 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 0 2020-03-11T22:37:47.985 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.985 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum": { 2020-03-11T22:37:47.985 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:37:47.985 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:37:47.985 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:37:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:37:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:37:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:37:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:37:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3179200, 2020-03-11T22:37:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 33152, 2020-03-11T22:37:47.986 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 13, 2020-03-11T22:37:47.987 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145714, 2020-03-11T22:37:47.987 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311393600, 2020-03-11T22:37:47.987 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:37:47.987 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:37:47.987 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318867046400, 2020-03-11T22:37:47.987 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:37:47.987 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 33947648, 2020-03-11T22:37:47.988 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 403228, 2020-03-11T22:37:47.988 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:37:47.988 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:47.988 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:37:47.988 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 14316, 2020-03-11T22:37:47.988 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221211156 2020-03-11T22:37:47.988 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.989 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:37:47.989 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:37:47.989 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:37:47.989 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:37:47.989 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:37:47.989 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:37:47.989 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:37:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:37:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:37:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:37:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:37:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:37:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.990 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:37:47.991 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:37:47.991 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.991 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum_by_class": { 2020-03-11T22:37:47.991 INFO:tasks.workunit.client.0.smithi016.stdout: "hdd": { 2020-03-11T22:37:47.991 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:37:47.991 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:37:47.991 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:37:47.992 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:37:47.992 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:37:47.992 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:37:47.992 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:37:47.992 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3179200, 2020-03-11T22:37:47.992 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 33152, 2020-03-11T22:37:47.992 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 13, 2020-03-11T22:37:47.993 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145714, 2020-03-11T22:37:47.993 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311393600, 2020-03-11T22:37:47.993 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:37:47.993 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:37:47.993 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318867046400, 2020-03-11T22:37:47.993 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:37:47.993 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 33947648, 2020-03-11T22:37:47.994 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 403228, 2020-03-11T22:37:47.994 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:37:47.994 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:47.994 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:37:47.994 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 14316, 2020-03-11T22:37:47.994 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221211156 2020-03-11T22:37:47.994 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.995 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:37:47.995 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:37:47.995 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:37:47.996 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:37:47.996 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:37:47.996 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:37:47.996 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:37:47.996 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.996 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:37:47.997 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:37:47.997 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:37:47.997 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:37:47.997 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:37:47.998 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.998 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:37:47.998 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:37:47.999 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:47.999 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:47.999 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_stats": [ 2020-03-11T22:37:47.999 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:47.999 INFO:tasks.workunit.client.0.smithi016.stdout: "poolid": 1, 2020-03-11T22:37:47.999 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:37:48.000 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:37:48.000 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48300, 2020-03-11T22:37:48.000 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:37:48.000 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:37:48.000 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:37:48.000 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:37:48.000 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:37:48.001 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:37:48.001 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:37:48.001 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:37:48.001 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:37:48.001 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:37:48.001 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:37:48.001 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:37:48.002 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 100, 2020-03-11T22:37:48.002 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 100, 2020-03-11T22:37:48.002 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:37:48.002 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:37:48.002 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:37:48.002 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:37:48.002 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:37:48.003 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:37:48.003 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:37:48.003 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:37:48.003 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:37:48.004 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:37:48.004 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:37:48.004 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:37:48.004 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:37:48.004 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:37:48.004 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:37:48.004 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:37:48.005 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:37:48.005 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:37:48.005 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:37:48.005 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:37:48.005 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:37:48.005 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:37:48.005 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:37:48.006 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:37:48.006 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:37:48.006 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:48.006 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:37:48.006 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:37:48.006 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:37:48.006 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:37:48.007 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 19529728, 2020-03-11T22:37:48.007 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 143228, 2020-03-11T22:37:48.007 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:37:48.007 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:48.007 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:37:48.007 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:37:48.007 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:37:48.008 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:48.008 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 16, 2020-03-11T22:37:48.008 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 16, 2020-03-11T22:37:48.008 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 2, 2020-03-11T22:37:48.008 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 2, 2020-03-11T22:37:48.008 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 3 2020-03-11T22:37:48.008 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:48.008 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:48.009 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_stats": [ 2020-03-11T22:37:48.009 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:48.009 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:37:48.009 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 150323855370 2020-03-11T22:37:48.009 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:48.009 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:48.009 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:37:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 42949672975 2020-03-11T22:37:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:37:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 154618822666 2020-03-11T22:37:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:48.010 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_state": [ 2020-03-11T22:37:48.011 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:48.011 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "active+undersized+degraded", 2020-03-11T22:37:48.011 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 1 2020-03-11T22:37:48.011 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:48.011 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:48.011 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_osd": [ 2020-03-11T22:37:48.011 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:48.012 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:37:48.012 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:37:48.012 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:37:48.012 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:48.012 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:48.012 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:48.012 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:37:48.012 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:37:48.013 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 0, 2020-03-11T22:37:48.013 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:48.013 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:48.013 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:48.013 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:37:48.013 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 1, 2020-03-11T22:37:48.013 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:37:48.014 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:48.014 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:48.014 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:48.014 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": [ 2020-03-11T22:37:48.014 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:37:48.014 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:37:48.014 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": {} 2020-03-11T22:37:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:37:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: "servicemap": { 2020-03-11T22:37:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:37:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:36:05.916502+0000", 2020-03-11T22:37:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: "services": {} 2020-03-11T22:37:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:37:48.015 INFO:tasks.workunit.client.0.smithi016.stdout: "paxos": { 2020-03-11T22:37:48.016 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:37:48.016 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 313, 2020-03-11T22:37:48.016 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pn": 0, 2020-03-11T22:37:48.016 INFO:tasks.workunit.client.0.smithi016.stdout: "accepted_pn": 0 2020-03-11T22:37:48.016 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:37:48.016 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:37:48.017 INFO:tasks.workunit.client.0.smithi016.stderr:report 3297779056 2020-03-11T22:37:48.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:37:48.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-11T22:37:48.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-11T22:37:48.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-11T22:37:48.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:48.019 INFO:tasks.workunit.client.0.smithi016.stdout:writing non-divergent object existing_4 2020-03-11T22:37:48.208 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:48.208 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:37:12.196947+0000 29'100 39:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:37:48.208 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:37:48.209 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:48.210 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:37:48.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:37:48.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:37:48.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-11T22:37:48.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:48.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:48.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:48.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:48.333 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:37:48.334 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:48.334 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:48.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:48.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:48.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:48.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:48.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:48.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:48.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:48.392 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:37:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:37:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:48.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855373 2020-03-11T22:37:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855373' 2020-03-11T22:37:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:48.892 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:37:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:37:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855373 1-' 2020-03-11T22:37:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:48.997 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:49.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822668 2020-03-11T22:37:49.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855373 1- 2-154618822668' 2020-03-11T22:37:49.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:49.133 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855373 2020-03-11T22:37:49.133 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:49.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:49.134 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855373 2020-03-11T22:37:49.134 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:49.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855373 2020-03-11T22:37:49.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855373' 2020-03-11T22:37:49.135 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 150323855373 2020-03-11T22:37:49.136 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:49.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855371 -lt 150323855373 2020-03-11T22:37:49.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:50.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:50.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:50.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855373 -lt 150323855373 2020-03-11T22:37:50.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:50.873 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:37:50.873 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:50.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:50.874 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:37:50.874 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:50.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:37:50.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:37:50.875 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 2020-03-11T22:37:50.876 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:51.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:37:51.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:37:51.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:51.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822668 2020-03-11T22:37:51.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:51.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:51.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822668 2020-03-11T22:37:51.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:51.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822668 2020-03-11T22:37:51.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822668' 2020-03-11T22:37:51.240 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:51.240 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 154618822668 2020-03-11T22:37:51.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822668 -lt 154618822668 2020-03-11T22:37:51.603 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:51.604 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:51.604 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:52.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:37:52.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:52.107 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:52.107 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:52.107 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:52.107 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:52.108 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:52.108 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:52.460 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:52.460 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:52.461 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:52.461 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:52.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:52.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:37:52.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:37:52.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:37:52.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:52.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:37:53.033 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:53.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:53.034 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:53.034 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:53.034 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:53.034 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:53.034 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:53.035 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:53.384 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:53.384 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:53.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:53.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:53.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:53.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:53.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:53.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:53.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:53.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:53.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:53.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:53.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:54.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:54.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:54.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:37:54.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:54.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:37:54.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:54.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:54.525 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:54.525 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:54.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:54.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:54.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:54.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:54.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:54.882 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:54.882 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:54.882 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:55.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:55.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:55.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:55.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:55.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:55.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:55.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:55.363 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:55.363 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:55.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:55.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:55.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:37:55.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:55.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:37:56.242 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:56.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:56.244 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:56.244 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:56.244 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:56.244 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:56.244 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:56.244 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:56.646 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:56.647 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:56.647 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:56.647 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:57.116 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:57.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:57.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:57.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:57.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:57.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:57.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:57.118 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:57.118 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:57.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:57.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:57.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:37:57.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:57.584 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:37:58.384 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:58.384 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:58.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:58.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:58.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:58.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:58.386 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:58.386 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:58.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:58.746 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:58.746 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:58.747 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:59.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:59.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:59.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:59.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:59.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:59.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:59.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:59.216 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:59.216 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:59.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:59.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:59.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:37:59.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:59.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:38:01.289 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:01.289 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:01.290 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:01.290 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:01.291 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:01.291 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:01.291 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:01.291 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:01.675 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:01.675 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:01.676 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:01.676 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:02.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:02.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:02.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:02.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:02.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:02.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:02.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:02.157 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:02.157 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:02.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:02.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:02.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:38:02.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:02.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:38:05.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:05.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:05.832 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:05.832 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:05.832 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:05.832 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:05.832 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:05.832 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:06.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:06.193 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:06.193 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:06.193 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:06.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:06.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:06.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:06.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:06.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:06.662 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:06.662 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:06.662 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:06.662 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:07.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:07.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:07.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:38:07.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:07.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:38:13.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:13.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:13.543 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:13.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:13.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:13.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:13.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:13.545 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:13.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:13.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:13.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:13.906 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:14.380 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:14.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:14.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:14.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:14.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:14.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:14.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:14.382 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:14.382 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:38:14.864 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:14.865 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:38:22.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:22.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:22.166 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:22.166 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:22.166 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:22.166 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:22.167 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:22.167 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:22.577 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:22.577 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:22.578 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:22.578 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:23.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:23.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:23.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:23.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:23.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:23.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:23.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:23.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:23.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:23.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:23.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:23.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:38:23.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:38:23.999 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:38:23.999 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_fingerprint": "cb131c3c-fd97-4878-9763-256d0f3d43f9", 2020-03-11T22:38:24.000 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:38:24.000 INFO:tasks.workunit.client.0.smithi016.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:38:24.000 INFO:tasks.workunit.client.0.smithi016.stdout: "timestamp": "2020-03-11T22:38:23.996126+0000", 2020-03-11T22:38:24.000 INFO:tasks.workunit.client.0.smithi016.stdout: "tag": "", 2020-03-11T22:38:24.000 INFO:tasks.workunit.client.0.smithi016.stdout: "health": { 2020-03-11T22:38:24.000 INFO:tasks.workunit.client.0.smithi016.stdout: "status": "HEALTH_WARN", 2020-03-11T22:38:24.001 INFO:tasks.workunit.client.0.smithi016.stdout: "checks": { 2020-03-11T22:38:24.001 INFO:tasks.workunit.client.0.smithi016.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:38:24.001 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:24.001 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:38:24.001 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:38:24.002 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 17 2020-03-11T22:38:24.002 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.002 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [], 2020-03-11T22:38:24.002 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:38:24.002 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.003 INFO:tasks.workunit.client.0.smithi016.stdout: "OSD_DOWN": { 2020-03-11T22:38:24.003 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:24.003 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:38:24.003 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 osds down", 2020-03-11T22:38:24.003 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:38:24.003 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.004 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:38:24.004 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.004 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "osd.1 (root=default,host=smithi016) is down" 2020-03-11T22:38:24.004 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.004 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.004 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:38:24.005 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.005 INFO:tasks.workunit.client.0.smithi016.stdout: "PG_DEGRADED": { 2020-03-11T22:38:24.005 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:24.005 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:38:24.005 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-11T22:38:24.005 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 2 2020-03-11T22:38:24.006 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.006 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:38:24.006 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.006 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.0 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:38:24.006 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.006 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.007 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:38:24.007 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.007 INFO:tasks.workunit.client.0.smithi016.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:38:24.007 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:24.007 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:38:24.007 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:38:24.008 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:38:24.008 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.008 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:38:24.008 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.009 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:38:24.009 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.009 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.009 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:38:24.009 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:38:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: "mutes": [] 2020-03-11T22:38:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.010 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_first_committed": 1, 2020-03-11T22:38:24.011 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_last_committed": 1, 2020-03-11T22:38:24.011 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap": { 2020-03-11T22:38:24.011 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:38:24.011 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:38:24.011 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:36:05.205097Z", 2020-03-11T22:38:24.011 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:36:05.205097Z", 2020-03-11T22:38:24.012 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release": 15, 2020-03-11T22:38:24.012 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:38:24.012 INFO:tasks.workunit.client.0.smithi016.stdout: "features": { 2020-03-11T22:38:24.012 INFO:tasks.workunit.client.0.smithi016.stdout: "persistent": [ 2020-03-11T22:38:24.012 INFO:tasks.workunit.client.0.smithi016.stdout: "kraken", 2020-03-11T22:38:24.012 INFO:tasks.workunit.client.0.smithi016.stdout: "luminous", 2020-03-11T22:38:24.012 INFO:tasks.workunit.client.0.smithi016.stdout: "mimic", 2020-03-11T22:38:24.013 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap-prune", 2020-03-11T22:38:24.013 INFO:tasks.workunit.client.0.smithi016.stdout: "nautilus", 2020-03-11T22:38:24.013 INFO:tasks.workunit.client.0.smithi016.stdout: "octopus" 2020-03-11T22:38:24.013 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.013 INFO:tasks.workunit.client.0.smithi016.stdout: "optional": [] 2020-03-11T22:38:24.013 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.014 INFO:tasks.workunit.client.0.smithi016.stdout: "mons": [ 2020-03-11T22:38:24.014 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.014 INFO:tasks.workunit.client.0.smithi016.stdout: "rank": 0, 2020-03-11T22:38:24.014 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "a", 2020-03-11T22:38:24.014 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:38:24.014 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.014 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.015 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.015 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:38:24.015 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 0 2020-03-11T22:38:24.015 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.015 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.015 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 0, 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 0 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.017 INFO:tasks.workunit.client.0.smithi016.stdout: "quorum": [ 2020-03-11T22:38:24.017 INFO:tasks.workunit.client.0.smithi016.stdout: 0 2020-03-11T22:38:24.017 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.017 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap": { 2020-03-11T22:38:24.017 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 39, 2020-03-11T22:38:24.017 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:38:24.018 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:36:05.916167+0000", 2020-03-11T22:38:24.022 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:37:12.083146+0000", 2020-03-11T22:38:24.022 INFO:tasks.workunit.client.0.smithi016.stdout: "last_up_change": "2020-03-11T22:37:10.313743+0000", 2020-03-11T22:38:24.022 INFO:tasks.workunit.client.0.smithi016.stdout: "last_in_change": "2020-03-11T22:37:11.863615+0000", 2020-03-11T22:38:24.022 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:38:24.023 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_num": 5800384, 2020-03-11T22:38:24.023 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_set": [ 2020-03-11T22:38:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: "nodown", 2020-03-11T22:38:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: "noin", 2020-03-11T22:38:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: "noout", 2020-03-11T22:38:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: "pglog_hardlimit", 2020-03-11T22:38:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snapdirs", 2020-03-11T22:38:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: "recovery_deletes", 2020-03-11T22:38:24.024 INFO:tasks.workunit.client.0.smithi016.stdout: "sortbitwise" 2020-03-11T22:38:24.025 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.025 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_version": 7, 2020-03-11T22:38:24.025 INFO:tasks.workunit.client.0.smithi016.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:38:24.025 INFO:tasks.workunit.client.0.smithi016.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:38:24.025 INFO:tasks.workunit.client.0.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:38:24.026 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_snapshot": "", 2020-03-11T22:38:24.026 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_max": 1, 2020-03-11T22:38:24.026 INFO:tasks.workunit.client.0.smithi016.stdout: "max_osd": 3, 2020-03-11T22:38:24.026 INFO:tasks.workunit.client.0.smithi016.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:38:24.026 INFO:tasks.workunit.client.0.smithi016.stdout: "min_compat_client": "jewel", 2020-03-11T22:38:24.026 INFO:tasks.workunit.client.0.smithi016.stdout: "require_osd_release": "octopus", 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi016.stdout: "pools": [ 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_name": "test", 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi016.stdout: "create_time": "2020-03-11T22:36:28.601035+0000", 2020-03-11T22:38:24.028 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": 1, 2020-03-11T22:38:24.028 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_names": "hashpspool", 2020-03-11T22:38:24.028 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:38:24.029 INFO:tasks.workunit.client.0.smithi016.stdout: "size": 3, 2020-03-11T22:38:24.029 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 2, 2020-03-11T22:38:24.029 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_rule": 0, 2020-03-11T22:38:24.029 INFO:tasks.workunit.client.0.smithi016.stdout: "object_hash": 2, 2020-03-11T22:38:24.029 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:38:24.029 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num": 1, 2020-03-11T22:38:24.030 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num": 1, 2020-03-11T22:38:24.030 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num_target": 1, 2020-03-11T22:38:24.030 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_target": 1, 2020-03-11T22:38:24.030 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_pending": 1, 2020-03-11T22:38:24.030 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pg_merge_meta": { 2020-03-11T22:38:24.030 INFO:tasks.workunit.client.0.smithi016.stdout: "source_pgid": "0.0", 2020-03-11T22:38:24.031 INFO:tasks.workunit.client.0.smithi016.stdout: "ready_epoch": 0, 2020-03-11T22:38:24.031 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_started": 0, 2020-03-11T22:38:24.031 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_clean": 0, 2020-03-11T22:38:24.031 INFO:tasks.workunit.client.0.smithi016.stdout: "source_version": "0'0", 2020-03-11T22:38:24.031 INFO:tasks.workunit.client.0.smithi016.stdout: "target_version": "0'0" 2020-03-11T22:38:24.031 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.032 INFO:tasks.workunit.client.0.smithi016.stdout: "last_change": "29", 2020-03-11T22:38:24.032 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend": "0", 2020-03-11T22:38:24.032 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:38:24.032 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:38:24.032 INFO:tasks.workunit.client.0.smithi016.stdout: "auid": 0, 2020-03-11T22:38:24.032 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:38:24.033 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_seq": 0, 2020-03-11T22:38:24.033 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_epoch": 0, 2020-03-11T22:38:24.033 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_snaps": [], 2020-03-11T22:38:24.033 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps": "[]", 2020-03-11T22:38:24.033 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_bytes": 0, 2020-03-11T22:38:24.033 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_objects": 0, 2020-03-11T22:38:24.033 INFO:tasks.workunit.client.0.smithi016.stdout: "tiers": [], 2020-03-11T22:38:24.034 INFO:tasks.workunit.client.0.smithi016.stdout: "tier_of": -1, 2020-03-11T22:38:24.034 INFO:tasks.workunit.client.0.smithi016.stdout: "read_tier": -1, 2020-03-11T22:38:24.034 INFO:tasks.workunit.client.0.smithi016.stdout: "write_tier": -1, 2020-03-11T22:38:24.034 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_mode": "none", 2020-03-11T22:38:24.034 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_bytes": 0, 2020-03-11T22:38:24.034 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_objects": 0, 2020-03-11T22:38:24.035 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:38:24.035 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:38:24.035 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:38:24.035 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_flush_age": 0, 2020-03-11T22:38:24.035 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_evict_age": 0, 2020-03-11T22:38:24.035 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profile": "", 2020-03-11T22:38:24.035 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_params": { 2020-03-11T22:38:24.036 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "none" 2020-03-11T22:38:24.036 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.036 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_period": 0, 2020-03-11T22:38:24.036 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_count": 0, 2020-03-11T22:38:24.036 INFO:tasks.workunit.client.0.smithi016.stdout: "use_gmt_hitset": true, 2020-03-11T22:38:24.036 INFO:tasks.workunit.client.0.smithi016.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:38:24.037 INFO:tasks.workunit.client.0.smithi016.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:38:24.037 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:38:24.037 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:38:24.037 INFO:tasks.workunit.client.0.smithi016.stdout: "grade_table": [], 2020-03-11T22:38:24.037 INFO:tasks.workunit.client.0.smithi016.stdout: "stripe_width": 0, 2020-03-11T22:38:24.038 INFO:tasks.workunit.client.0.smithi016.stdout: "expected_num_objects": 0, 2020-03-11T22:38:24.039 INFO:tasks.workunit.client.0.smithi016.stdout: "fast_read": false, 2020-03-11T22:38:24.039 INFO:tasks.workunit.client.0.smithi016.stdout: "options": {}, 2020-03-11T22:38:24.039 INFO:tasks.workunit.client.0.smithi016.stdout: "application_metadata": {} 2020-03-11T22:38:24.039 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.039 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.040 INFO:tasks.workunit.client.0.smithi016.stdout: "osds": [ 2020-03-11T22:38:24.040 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.040 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:38:24.040 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "af406440-1a7b-405f-b1f1-ab2ed8bfbc14", 2020-03-11T22:38:24.040 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:38:24.040 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:38:24.040 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:38:24.041 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:38:24.041 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 5, 2020-03-11T22:38:24.041 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 29, 2020-03-11T22:38:24.041 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 35, 2020-03-11T22:38:24.041 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 38, 2020-03-11T22:38:24.041 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 30, 2020-03-11T22:38:24.042 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:38:24.042 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:38:24.042 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.042 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.042 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.042 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:38:24.043 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:38:24.043 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.043 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.043 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.043 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:38:24.043 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:38:24.044 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.044 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.044 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.044 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:38:24.044 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.044 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.045 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.045 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:38:24.045 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:38:24.045 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.045 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.045 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.046 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:38:24.046 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:38:24.046 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.046 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.046 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.046 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:24.046 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.046 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.047 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.047 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:38:24.047 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:38:24.047 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.047 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.048 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.048 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:38:24.048 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:38:24.048 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.048 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.048 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.048 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:24.049 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.049 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.049 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.049 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:38:24.050 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:38:24.050 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.050 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.050 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.050 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:38:24.050 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30308 2020-03-11T22:38:24.051 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.051 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.051 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.051 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6803/30308", 2020-03-11T22:38:24.051 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6805/30308", 2020-03-11T22:38:24.051 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30308", 2020-03-11T22:38:24.052 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30308", 2020-03-11T22:38:24.052 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:38:24.052 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:38:24.052 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:38:24.052 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.052 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.052 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.053 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:38:24.053 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "f4026fd4-5a11-42e2-b606-f350d010eed0", 2020-03-11T22:38:24.053 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 0, 2020-03-11T22:38:24.053 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:38:24.053 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:38:24.053 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:38:24.054 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 0, 2020-03-11T22:38:24.054 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 0, 2020-03-11T22:38:24.054 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 10, 2020-03-11T22:38:24.054 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 23, 2020-03-11T22:38:24.054 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 31, 2020-03-11T22:38:24.055 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:38:24.055 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:38:24.055 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.055 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.055 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.055 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:38:24.055 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:38:24.056 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.056 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.056 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.056 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:38:24.056 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:38:24.057 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.057 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.057 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.058 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:38:24.058 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.058 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.058 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.058 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:38:24.058 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:38:24.059 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.059 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.059 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.059 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:38:24.059 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:38:24.060 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.060 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.060 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.060 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:24.060 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.061 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.061 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.061 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:38:24.061 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:38:24.061 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.061 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.061 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.062 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:38:24.062 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:38:24.062 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.062 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.062 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.063 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:24.063 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.063 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.063 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.064 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:38:24.064 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:38:24.065 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.065 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.065 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.065 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:38:24.065 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 25904 2020-03-11T22:38:24.066 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.066 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.066 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.066 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/25904", 2020-03-11T22:38:24.066 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/25904", 2020-03-11T22:38:24.066 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25904", 2020-03-11T22:38:24.067 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25904", 2020-03-11T22:38:24.067 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:38:24.067 INFO:tasks.workunit.client.0.smithi016.stdout: "exists" 2020-03-11T22:38:24.067 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.067 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.068 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.068 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:38:24.068 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "8b8190ff-543f-4f23-9247-1ee5e94a2b36", 2020-03-11T22:38:24.068 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:38:24.068 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:38:24.068 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:38:24.069 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:38:24.069 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 15, 2020-03-11T22:38:24.069 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 29, 2020-03-11T22:38:24.069 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 36, 2020-03-11T22:38:24.069 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 0, 2020-03-11T22:38:24.069 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 32, 2020-03-11T22:38:24.069 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:38:24.070 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:38:24.070 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.070 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.071 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.071 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:38:24.071 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:38:24.071 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.071 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.071 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.072 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:38:24.072 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:38:24.072 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.073 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.073 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.073 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:38:24.073 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.073 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.073 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.073 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:38:24.074 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:38:24.074 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.074 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.074 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.074 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.077 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.077 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:38:24.077 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:38:24.077 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.077 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.077 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.077 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:24.077 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:38:24.078 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.078 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:38:24.078 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:38:24.078 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:38:24.078 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.078 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.079 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:38:24.079 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:38:24.079 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 30502 2020-03-11T22:38:24.079 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.079 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.079 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.080 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/30502", 2020-03-11T22:38:24.080 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/30502", 2020-03-11T22:38:24.080 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30502", 2020-03-11T22:38:24.080 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30502", 2020-03-11T22:38:24.080 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:38:24.081 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:38:24.081 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:38:24.081 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.081 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.081 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.081 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_xinfo": [ 2020-03-11T22:38:24.082 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.082 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:38:24.082 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:37:04.006421+0000", 2020-03-11T22:38:24.082 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:38:24.082 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:38:24.082 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:38:24.083 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:38:24.083 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:13.035805+0000", 2020-03-11T22:38:24.083 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:38:24.083 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.083 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.083 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:38:24.083 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:37:04.560676+0000", 2020-03-11T22:38:24.084 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:38:24.084 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:38:24.084 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:38:24.084 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:38:24.084 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:19.439160+0000", 2020-03-11T22:38:24.084 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:38:24.084 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.085 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.085 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:38:24.085 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:37:05.072866+0000", 2020-03-11T22:38:24.085 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:38:24.085 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:38:24.085 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:38:24.086 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:38:24.086 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:26.266667+0000", 2020-03-11T22:38:24.086 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:38:24.086 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.087 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.087 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap": [], 2020-03-11T22:38:24.087 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap_items": [], 2020-03-11T22:38:24.087 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_temp": [], 2020-03-11T22:38:24.087 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_temp": [], 2020-03-11T22:38:24.087 INFO:tasks.workunit.client.0.smithi016.stdout: "blacklist": {}, 2020-03-11T22:38:24.087 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profiles": { 2020-03-11T22:38:24.088 INFO:tasks.workunit.client.0.smithi016.stdout: "default": { 2020-03-11T22:38:24.088 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:38:24.088 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "2", 2020-03-11T22:38:24.088 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:38:24.088 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van" 2020-03-11T22:38:24.088 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.088 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.089 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps_queue": [], 2020-03-11T22:38:24.089 INFO:tasks.workunit.client.0.smithi016.stdout: "new_removed_snaps": [], 2020-03-11T22:38:24.089 INFO:tasks.workunit.client.0.smithi016.stdout: "new_purged_snaps": [], 2020-03-11T22:38:24.089 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_node_flags": {}, 2020-03-11T22:38:24.090 INFO:tasks.workunit.client.0.smithi016.stdout: "device_class_flags": {} 2020-03-11T22:38:24.090 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.090 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_metadata": [ 2020-03-11T22:38:24.090 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.090 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:38:24.091 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:38:24.092 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6804/30308,v1:127.0.0.1:6805/30308]", 2020-03-11T22:38:24.092 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:38:24.092 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:38:24.092 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:24.092 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:24.092 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:24.093 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:24.093 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:24.093 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:24.093 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:24.093 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:38:24.093 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:24.093 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:24.094 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:24.094 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:24.094 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:38:24.094 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:24.094 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:24.094 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:38:24.094 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:38:24.095 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:38:24.095 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:24.095 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:38:24.095 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:38:24.095 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:24.095 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:38:24.095 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6802/30308,v1:127.0.0.1:6803/30308]", 2020-03-11T22:38:24.096 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:38:24.096 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30308,v1:127.0.0.1:6809/30308]", 2020-03-11T22:38:24.096 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30308,v1:127.0.0.1:6807/30308]", 2020-03-11T22:38:24.096 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:38:24.096 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:38:24.096 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:24.096 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:24.097 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:24.097 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:38:24.097 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:24.097 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:24.097 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:38:24.097 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:38:24.098 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:24.098 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:38:24.098 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.098 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.098 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:38:24.098 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:38:24.098 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/25904,v1:127.0.0.1:6813/25904]", 2020-03-11T22:38:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:38:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:38:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:24.100 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:24.100 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:38:24.100 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:24.100 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:24.100 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:24.100 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:24.100 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:38:24.101 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:24.101 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:24.101 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:38:24.101 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:38:24.101 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:38:24.101 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:24.101 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:38:24.102 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:38:24.102 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:24.102 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:38:24.102 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/25904,v1:127.0.0.1:6811/25904]", 2020-03-11T22:38:24.102 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:38:24.102 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25904,v1:127.0.0.1:6817/25904]", 2020-03-11T22:38:24.102 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25904,v1:127.0.0.1:6815/25904]", 2020-03-11T22:38:24.102 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:38:24.103 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:38:24.103 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:24.103 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:24.103 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:24.103 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:38:24.103 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:24.104 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:24.104 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:38:24.104 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:38:24.104 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:24.104 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:38:24.105 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.105 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.105 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:38:24.105 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:38:24.105 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/30502,v1:127.0.0.1:6813/30502]", 2020-03-11T22:38:24.105 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:38:24.105 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:38:24.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:24.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:24.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:24.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:24.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:24.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:24.106 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:24.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:38:24.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:24.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:24.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:24.107 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:24.107 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:38:24.107 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:24.107 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:24.108 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:38:24.108 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:38:24.108 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:38:24.108 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:24.108 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:38:24.108 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:38:24.108 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:24.109 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:38:24.109 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/30502,v1:127.0.0.1:6811/30502]", 2020-03-11T22:38:24.109 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:38:24.109 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30502,v1:127.0.0.1:6817/30502]", 2020-03-11T22:38:24.109 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30502,v1:127.0.0.1:6815/30502]", 2020-03-11T22:38:24.109 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:38:24.109 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:38:24.110 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:24.110 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:24.110 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:24.110 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:38:24.110 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:24.110 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:24.110 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:38:24.111 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:38:24.111 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:24.111 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:38:24.111 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.111 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.111 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_first_committed": 1, 2020-03-11T22:38:24.111 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_last_committed": 39, 2020-03-11T22:38:24.111 INFO:tasks.workunit.client.0.smithi016.stdout: "crushmap": { 2020-03-11T22:38:24.112 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": [ 2020-03-11T22:38:24.112 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.112 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:38:24.112 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.0", 2020-03-11T22:38:24.112 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:38:24.112 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.112 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.112 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:38:24.113 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.1", 2020-03-11T22:38:24.113 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:38:24.113 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.113 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.113 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:38:24.113 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.2", 2020-03-11T22:38:24.113 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:38:24.114 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.114 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.114 INFO:tasks.workunit.client.0.smithi016.stdout: "types": [ 2020-03-11T22:38:24.114 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.114 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 0, 2020-03-11T22:38:24.114 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd" 2020-03-11T22:38:24.114 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.114 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.115 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:38:24.115 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "host" 2020-03-11T22:38:24.116 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.117 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.117 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 2, 2020-03-11T22:38:24.117 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "chassis" 2020-03-11T22:38:24.117 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.117 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.118 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 3, 2020-03-11T22:38:24.118 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "rack" 2020-03-11T22:38:24.118 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.118 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.118 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 4, 2020-03-11T22:38:24.118 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "row" 2020-03-11T22:38:24.118 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.119 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.119 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 5, 2020-03-11T22:38:24.119 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pdu" 2020-03-11T22:38:24.119 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.119 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.119 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 6, 2020-03-11T22:38:24.119 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pod" 2020-03-11T22:38:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 7, 2020-03-11T22:38:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "room" 2020-03-11T22:38:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 8, 2020-03-11T22:38:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "datacenter" 2020-03-11T22:38:24.121 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.121 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.121 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 9, 2020-03-11T22:38:24.121 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "zone" 2020-03-11T22:38:24.121 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.121 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.121 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 10, 2020-03-11T22:38:24.122 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "region" 2020-03-11T22:38:24.122 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.122 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.122 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:38:24.122 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "root" 2020-03-11T22:38:24.122 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.122 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.123 INFO:tasks.workunit.client.0.smithi016.stdout: "buckets": [ 2020-03-11T22:38:24.123 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.123 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -1, 2020-03-11T22:38:24.123 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default", 2020-03-11T22:38:24.123 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:38:24.123 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:38:24.124 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:38:24.124 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:38:24.124 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:38:24.124 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:38:24.124 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.124 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:38:24.125 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:38:24.125 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:38:24.125 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.125 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.125 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.125 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.125 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -2, 2020-03-11T22:38:24.126 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default~hdd", 2020-03-11T22:38:24.126 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:38:24.126 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:38:24.126 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:38:24.126 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:38:24.127 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016", 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:38:24.128 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:38:24.129 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:38:24.129 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:38:24.129 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:38:24.129 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:38:24.129 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.129 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:38:24.129 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:38:24.130 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:38:24.130 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.130 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.130 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:38:24.130 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:38:24.130 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:38:24.130 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.130 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.131 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:38:24.131 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:38:24.131 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:38:24.131 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.131 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.131 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.131 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.132 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:38:24.132 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016~hdd", 2020-03-11T22:38:24.132 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:38:24.132 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:38:24.132 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:38:24.132 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:38:24.132 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:38:24.133 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:38:24.133 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.133 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:38:24.133 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:38:24.133 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:38:24.133 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.133 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.134 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:38:24.134 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:38:24.134 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:38:24.134 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.134 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.134 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:38:24.134 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:38:24.134 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:38:24.135 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.135 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.135 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.135 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.135 INFO:tasks.workunit.client.0.smithi016.stdout: "rules": [ 2020-03-11T22:38:24.135 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.135 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 0, 2020-03-11T22:38:24.136 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "replicated_rule", 2020-03-11T22:38:24.136 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 0, 2020-03-11T22:38:24.136 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:38:24.136 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 1, 2020-03-11T22:38:24.136 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 10, 2020-03-11T22:38:24.136 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:38:24.136 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.137 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:38:24.137 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:38:24.137 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:38:24.137 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.137 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.137 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_firstn", 2020-03-11T22:38:24.138 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:38:24.138 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:38:24.138 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.138 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.138 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:38:24.138 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.138 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:38:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: "tunables": { 2020-03-11T22:38:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_tries": 0, 2020-03-11T22:38:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:38:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_total_tries": 50, 2020-03-11T22:38:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:38:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:38:24.140 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_stable": 1, 2020-03-11T22:38:24.140 INFO:tasks.workunit.client.0.smithi016.stdout: "straw_calc_version": 1, 2020-03-11T22:38:24.140 INFO:tasks.workunit.client.0.smithi016.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:38:24.140 INFO:tasks.workunit.client.0.smithi016.stdout: "profile": "jewel", 2020-03-11T22:38:24.140 INFO:tasks.workunit.client.0.smithi016.stdout: "optimal_tunables": 1, 2020-03-11T22:38:24.140 INFO:tasks.workunit.client.0.smithi016.stdout: "legacy_tunables": 0, 2020-03-11T22:38:24.140 INFO:tasks.workunit.client.0.smithi016.stdout: "minimum_required_version": "jewel", 2020-03-11T22:38:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables": 1, 2020-03-11T22:38:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables2": 1, 2020-03-11T22:38:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v2_rules": 0, 2020-03-11T22:38:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables3": 1, 2020-03-11T22:38:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v3_rules": 0, 2020-03-11T22:38:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v4_buckets": 1, 2020-03-11T22:38:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables5": 1, 2020-03-11T22:38:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v5_rules": 0 2020-03-11T22:38:24.142 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.142 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_args": {} 2020-03-11T22:38:24.142 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.142 INFO:tasks.workunit.client.0.smithi016.stdout: "fsmap": { 2020-03-11T22:38:24.142 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:38:24.142 INFO:tasks.workunit.client.0.smithi016.stdout: "default_fscid": -1, 2020-03-11T22:38:24.142 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": { 2020-03-11T22:38:24.143 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": {}, 2020-03-11T22:38:24.143 INFO:tasks.workunit.client.0.smithi016.stdout: "ro_compat": {}, 2020-03-11T22:38:24.143 INFO:tasks.workunit.client.0.smithi016.stdout: "incompat": { 2020-03-11T22:38:24.143 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_1": "base v0.20", 2020-03-11T22:38:24.143 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:38:24.143 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:38:24.143 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:38:24.144 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:38:24.144 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:38:24.144 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_8": "no anchor table", 2020-03-11T22:38:24.144 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_9": "file layout v2", 2020-03-11T22:38:24.144 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:38:24.144 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.144 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.145 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_flags": { 2020-03-11T22:38:24.145 INFO:tasks.workunit.client.0.smithi016.stdout: "enable_multiple": false, 2020-03-11T22:38:24.145 INFO:tasks.workunit.client.0.smithi016.stdout: "ever_enabled_multiple": false 2020-03-11T22:38:24.145 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.145 INFO:tasks.workunit.client.0.smithi016.stdout: "standbys": [], 2020-03-11T22:38:24.145 INFO:tasks.workunit.client.0.smithi016.stdout: "filesystems": [] 2020-03-11T22:38:24.145 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.146 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:38:24.146 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:38:24.146 INFO:tasks.workunit.client.0.smithi016.stdout: "auth": { 2020-03-11T22:38:24.146 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:38:24.146 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 5, 2020-03-11T22:38:24.146 INFO:tasks.workunit.client.0.smithi016.stdout: "num_secrets": 3 2020-03-11T22:38:24.147 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.147 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:38:24.147 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_active": 1, 2020-03-11T22:38:24.147 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_unknown": 0, 2020-03-11T22:38:24.147 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osd": 3, 2020-03-11T22:38:24.147 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_sum": { 2020-03-11T22:38:24.147 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:38:24.148 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 47947, 2020-03-11T22:38:24.148 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:38:24.148 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:38:24.148 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:38:24.148 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:38:24.148 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:38:24.148 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:38:24.149 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:38:24.149 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:38:24.149 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:38:24.149 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:38:24.149 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:38:24.149 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:38:24.149 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 101, 2020-03-11T22:38:24.149 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 101, 2020-03-11T22:38:24.150 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:38:24.150 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:38:24.150 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:38:24.150 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:38:24.150 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:38:24.150 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:38:24.150 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:38:24.151 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:38:24.151 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:38:24.151 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:38:24.151 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:38:24.151 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:38:24.151 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:38:24.151 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:38:24.151 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:38:24.152 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:38:24.152 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:38:24.152 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:38:24.152 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:38:24.152 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:38:24.152 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:38:24.152 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:38:24.153 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:38:24.153 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:38:24.153 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:38:24.153 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.153 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:38:24.153 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:38:24.154 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:38:24.154 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:38:24.154 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 0, 2020-03-11T22:38:24.154 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 0, 2020-03-11T22:38:24.154 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:38:24.154 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:24.154 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:38:24.155 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:38:24.155 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:38:24.155 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.155 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 17, 2020-03-11T22:38:24.155 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 17, 2020-03-11T22:38:24.155 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 2, 2020-03-11T22:38:24.155 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 2, 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 0 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum": { 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:38:24.157 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:38:24.157 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:38:24.157 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3179200, 2020-03-11T22:38:24.157 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 33152, 2020-03-11T22:38:24.157 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 13, 2020-03-11T22:38:24.157 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145714, 2020-03-11T22:38:24.157 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311393600, 2020-03-11T22:38:24.158 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:38:24.158 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:38:24.158 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318867046400, 2020-03-11T22:38:24.158 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:38:24.158 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 33947648, 2020-03-11T22:38:24.158 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 402522, 2020-03-11T22:38:24.158 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:38:24.159 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:24.159 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:38:24.159 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 14316, 2020-03-11T22:38:24.159 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221211156 2020-03-11T22:38:24.159 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.159 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:38:24.159 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:38:24.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:38:24.160 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:38:24.160 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:38:24.160 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:38:24.160 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:38:24.160 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.160 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:38:24.160 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:38:24.161 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:38:24.161 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:38:24.161 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:38:24.161 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.161 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:38:24.161 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:38:24.161 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.162 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum_by_class": { 2020-03-11T22:38:24.162 INFO:tasks.workunit.client.0.smithi016.stdout: "hdd": { 2020-03-11T22:38:24.162 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:38:24.162 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:38:24.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:38:24.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:38:24.162 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:38:24.163 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:38:24.163 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:38:24.163 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3179200, 2020-03-11T22:38:24.163 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 33152, 2020-03-11T22:38:24.163 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 13, 2020-03-11T22:38:24.163 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145714, 2020-03-11T22:38:24.163 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311393600, 2020-03-11T22:38:24.163 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:38:24.164 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:38:24.164 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318867046400, 2020-03-11T22:38:24.164 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:38:24.164 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 33947648, 2020-03-11T22:38:24.164 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 402522, 2020-03-11T22:38:24.164 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:38:24.164 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:24.165 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:38:24.165 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 14316, 2020-03-11T22:38:24.165 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221211156 2020-03-11T22:38:24.165 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.165 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:38:24.165 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:38:24.165 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:38:24.166 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:38:24.166 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:38:24.166 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:38:24.166 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:38:24.166 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.166 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:38:24.166 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:38:24.167 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:38:24.167 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:38:24.167 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:38:24.167 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.167 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:38:24.167 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:38:24.167 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.167 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.168 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_stats": [ 2020-03-11T22:38:24.168 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.168 INFO:tasks.workunit.client.0.smithi016.stdout: "poolid": 1, 2020-03-11T22:38:24.168 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:38:24.168 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:38:24.168 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 47947, 2020-03-11T22:38:24.168 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:38:24.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:38:24.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:38:24.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:38:24.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:38:24.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:38:24.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:38:24.169 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:38:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:38:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:38:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:38:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:38:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 101, 2020-03-11T22:38:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 101, 2020-03-11T22:38:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:38:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:38:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:38:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:38:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:38:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:38:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:38:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:38:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:38:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:38:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:38:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:38:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:38:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:38:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:38:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:38:24.173 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:38:24.173 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:38:24.173 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:38:24.173 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:38:24.173 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:38:24.173 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:38:24.174 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:38:24.174 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:38:24.174 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:38:24.174 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.174 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:38:24.174 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:38:24.174 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:38:24.174 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:38:24.175 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 19529728, 2020-03-11T22:38:24.175 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 142522, 2020-03-11T22:38:24.175 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:38:24.175 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:24.175 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:38:24.175 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:38:24.175 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:38:24.176 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.176 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 17, 2020-03-11T22:38:24.176 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 17, 2020-03-11T22:38:24.176 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 2, 2020-03-11T22:38:24.176 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 2, 2020-03-11T22:38:24.176 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 3 2020-03-11T22:38:24.176 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.176 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.177 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_stats": [ 2020-03-11T22:38:24.177 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.177 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:38:24.177 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 150323855379 2020-03-11T22:38:24.177 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.177 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.177 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:38:24.177 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 42949672975 2020-03-11T22:38:24.178 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.178 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.178 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:38:24.178 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 154618822675 2020-03-11T22:38:24.178 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.178 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.178 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_state": [ 2020-03-11T22:38:24.178 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.179 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "active+undersized+degraded", 2020-03-11T22:38:24.179 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 1 2020-03-11T22:38:24.179 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.179 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.179 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_osd": [ 2020-03-11T22:38:24.179 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.179 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:38:24.180 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:38:24.180 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:38:24.180 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:24.180 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.180 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.180 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:38:24.180 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:38:24.180 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 0, 2020-03-11T22:38:24.181 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:24.181 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.181 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.181 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:38:24.181 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 1, 2020-03-11T22:38:24.181 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:38:24.181 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:24.182 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.182 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.182 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": [ 2020-03-11T22:38:24.182 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:38:24.182 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:38:24.182 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": {} 2020-03-11T22:38:24.182 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.182 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:38:24.183 INFO:tasks.workunit.client.0.smithi016.stdout: "servicemap": { 2020-03-11T22:38:24.183 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:38:24.183 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:36:05.916502+0000", 2020-03-11T22:38:24.183 INFO:tasks.workunit.client.0.smithi016.stdout: "services": {} 2020-03-11T22:38:24.183 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:38:24.183 INFO:tasks.workunit.client.0.smithi016.stdout: "paxos": { 2020-03-11T22:38:24.183 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:38:24.184 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 390, 2020-03-11T22:38:24.184 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pn": 0, 2020-03-11T22:38:24.184 INFO:tasks.workunit.client.0.smithi016.stdout: "accepted_pn": 0 2020-03-11T22:38:24.184 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:38:24.184 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:38:24.185 INFO:tasks.workunit.client.0.smithi016.stderr:report 3469261495 2020-03-11T22:38:24.185 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:38:24.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-11T22:38:24.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:24.186 INFO:tasks.workunit.client.0.smithi016.stdout:delay recovery 2020-03-11T22:38:24.353 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:24.377 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:12.196947+0000 39'101 39:121 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:38:24.378 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:38:24.378 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:24.379 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:38:24.380 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:24.380 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:38:24.380 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:38:24.380 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:38:24.380 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:24.380 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:24.381 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:38:24.381 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.0.asok 2020-03-11T22:38:24.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:24.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:38:24.460 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:24.478 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:24.479 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:38:24.479 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:38:24.479 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:38:24.479 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:24.479 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:24.479 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:38:24.480 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.2.asok 2020-03-11T22:38:24.480 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:24.480 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:38:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:24.573 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-11T22:38:24.573 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:24.573 INFO:tasks.workunit.client.0.smithi016.stdout:revive divergent 1 2020-03-11T22:38:24.912 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:24.912 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:12.196947+0000 39'101 39:121 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:38:24.912 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:38:24.912 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:24.913 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:38:24.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-11T22:38:25.670 INFO:tasks.workunit.client.0.smithi016.stderr:noup is set 2020-03-11T22:38:25.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:38:25.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:25.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:25.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:38:25.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:25.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:25.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:25.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:25.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:25.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:25.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:25.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:25.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:25.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:25.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:25.683 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:25.683 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:25.684 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:25.684 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:25.684 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:25.684 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:38:25.684 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:38:25.684 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:38:25.684 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:25.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:25.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:25.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:25.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:25.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:25.685 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:25.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:25.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:25.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:38:25.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:25.686 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:38:25.687 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:38:25.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:38:25.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:25.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:25.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:25.709 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:25.706+0000 7f3dc47d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:25.711 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:25.708+0000 7f3dc47d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:25.712 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:25.709+0000 7f3dc47d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:26.047 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:38:26.047 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-11T22:38:26.231 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:26.228+0000 7f3dc47d4ec0 -1 Falling back to public interface 2020-03-11T22:38:27.147 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:27.143+0000 7f3dc47d4ec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:38:31.048 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-11T22:38:31.048 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:31.049 INFO:tasks.workunit.client.0.smithi016.stdout:delay recovery divergent 2020-03-11T22:38:31.400 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:31.400 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:12.196947+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:38:31.400 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:38:31.400 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:31.401 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:38:31.410 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:38:31.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:38:31.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:38:31.411 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:31.411 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:31.411 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:38:31.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.1.asok 2020-03-11T22:38:31.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:31.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:38:31.507 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:31.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-11T22:38:32.261 INFO:tasks.workunit.client.0.smithi016.stderr:noup is unset 2020-03-11T22:38:32.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-11T22:38:32.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:32.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:38:32.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:32.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:32.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:32.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:32.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:32.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:32.271 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:38:32.638 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/30308,v1:127.0.0.1:6803/30308] [v2:127.0.0.1:6804/30308,v1:127.0.0.1:6805/30308] exists,up af406440-1a7b-405f-b1f1-ab2ed8bfbc14 2020-03-11T22:38:32.639 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:38:32.639 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:32.639 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:32.639 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:32.639 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:38:32.639 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:32.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:32.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:32.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:32.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:32.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:32.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:32.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:33.001 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/33049,v1:127.0.0.1:6819/33049] [v2:127.0.0.1:6820/33049,v1:127.0.0.1:6821/33049] exists,up f4026fd4-5a11-42e2-b606-f350d010eed0 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:33.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:33.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:33.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:33.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:33.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:33.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:33.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:33.409 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/30502,v1:127.0.0.1:6811/30502] [v2:127.0.0.1:6812/30502,v1:127.0.0.1:6813/30502] exists,up 8b8190ff-543f-4f23-9247-1ee5e94a2b36 2020-03-11T22:38:33.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:33.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:33.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:33.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:33.783 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:33.784 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:12.196947+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:38:33.784 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:38:33.784 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:33.785 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:38:33.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-11T22:38:33.794 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:33.794 INFO:tasks.workunit.client.0.smithi016.stdout:wait for peering 2020-03-11T22:38:34.133 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:34.133 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:12.196947+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:38:34.133 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:38:34.134 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:34.134 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:38:34.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-11T22:38:40.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-11T22:38:40.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:40.237 INFO:tasks.workunit.client.0.smithi016.stdout:killing divergent 1 2020-03-11T22:38:40.587 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:38:32.568050+0000 39'101 45:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:38:40.588 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:40.589 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:38:40.597 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:38:40.598 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:40.598 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:40.598 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:40.598 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:40.598 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:40.703 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:40.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.17411.out 2020-03-11T22:38:40.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17411.out 2020-03-11T22:38:40.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:38:40.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:38:40.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:38:40.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:38:40.704 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:38:40.705 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17411.out 2020-03-11T22:38:41.312 INFO:tasks.workunit.client.0.smithi016.stderr:Exporting 1.0 info 1.0( v 45'102 lc 0'0 (29'90,45'102] local-lis/les=43/45 n=101 ec=23/23 lis/c=43/23 les/c/f=45/24/0 sis=43 pruub=7.632552175s) 2020-03-11T22:38:41.325 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:38:41.345 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:38:41.356 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:38:41.361 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-11T22:38:41.362 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-11T22:38:41.368 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-11T22:38:41.368 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-11T22:38:41.373 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:169157fa:::existing_81:head# 2020-03-11T22:38:41.377 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:17731382:::existing_58:head# 2020-03-11T22:38:41.378 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-11T22:38:41.379 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-11T22:38:41.385 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-11T22:38:41.386 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:28566edb:::existing_78:head# 2020-03-11T22:38:41.393 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:2a101182:::existing_20:head# 2020-03-11T22:38:41.394 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-11T22:38:41.394 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-11T22:38:41.396 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-11T22:38:41.397 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-11T22:38:41.397 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-11T22:38:41.397 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-11T22:38:41.407 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-11T22:38:41.409 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-11T22:38:41.417 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-11T22:38:41.421 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-11T22:38:41.427 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:53588912:::existing_9:head# 2020-03-11T22:38:41.428 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-11T22:38:41.428 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-11T22:38:41.429 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-11T22:38:41.429 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-11T22:38:41.434 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:602f83fe:::foo:head# 2020-03-11T22:38:41.439 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-11T22:38:41.439 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:628918ca:::existing_79:head# 2020-03-11T22:38:41.440 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-11T22:38:41.440 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-11T22:38:41.450 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-11T22:38:41.457 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-11T22:38:41.458 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-11T22:38:41.459 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-11T22:38:41.459 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-11T22:38:41.462 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-11T22:38:41.463 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:750ace21:::existing_19:head# 2020-03-11T22:38:41.463 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-11T22:38:41.464 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:79881152:::existing_63:head# 2020-03-11T22:38:41.464 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-11T22:38:41.464 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:81490677:::existing_76:head# 2020-03-11T22:38:41.465 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-11T22:38:41.465 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-11T22:38:41.466 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:38:41.466 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-11T22:38:41.476 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:92731e71:::existing_89:head# 2020-03-11T22:38:41.477 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:928205cf:::existing_82:head# 2020-03-11T22:38:41.477 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:93be2267:::existing_24:head# 2020-03-11T22:38:41.478 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-11T22:38:41.478 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-11T22:38:41.479 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-11T22:38:41.479 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-11T22:38:41.479 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-11T22:38:41.480 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:a504a533:::existing_90:head# 2020-03-11T22:38:41.480 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:a691337e:::existing_61:head# 2020-03-11T22:38:41.481 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:38:41.484 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-11T22:38:41.484 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-11T22:38:41.490 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-11T22:38:41.491 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-11T22:38:41.491 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:b343e277:::existing_88:head# 2020-03-11T22:38:41.492 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-11T22:38:41.494 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-11T22:38:41.494 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-11T22:38:41.495 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-11T22:38:41.497 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:bebec877:::existing_74:head# 2020-03-11T22:38:41.498 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-11T22:38:41.499 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:c156033e:::existing_100:head# 2020-03-11T22:38:41.499 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-11T22:38:41.499 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:38:41.499 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:38:41.501 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-11T22:38:41.502 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-11T22:38:41.502 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-11T22:38:41.502 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-11T22:38:41.502 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-11T22:38:41.502 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-11T22:38:41.502 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:38:41.502 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-11T22:38:41.503 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-11T22:38:41.503 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-11T22:38:41.503 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-11T22:38:41.503 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:e255b710:::existing_55:head# 2020-03-11T22:38:41.503 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-11T22:38:41.505 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-11T22:38:41.505 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-11T22:38:41.505 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:f6809456:::existing_11:head# 2020-03-11T22:38:41.505 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-11T22:38:41.506 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-11T22:38:41.506 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-11T22:38:41.506 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-11T22:38:41.506 INFO:tasks.workunit.client.0.smithi016.stderr:Export successful 2020-03-11T22:38:41.506 INFO:tasks.workunit.client.0.smithi016.stdout: marking collection for removal 2020-03-11T22:38:41.506 INFO:tasks.workunit.client.0.smithi016.stdout:setting '_remove' omap key 2020-03-11T22:38:41.506 INFO:tasks.workunit.client.0.smithi016.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:38:41.578 INFO:tasks.workunit.client.0.smithi016.stderr:Remove successful 2020-03-11T22:38:42.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17411.out 2020-03-11T22:38:42.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:38:42.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:38:42.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:38:42.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:38:42.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:38:42.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17411.out 2020-03-11T22:38:42.498 INFO:tasks.workunit.client.0.smithi016.stdout:get_pg_num_history pg_num_history pg_num_history(e45 pg_nums {1={23=1}} deleted_pools ) 2020-03-11T22:38:42.498 INFO:tasks.workunit.client.0.smithi016.stdout:Importing pgid 1.0 2020-03-11T22:38:42.499 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:38:42.499 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.585 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:38:42.585 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.593 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:38:42.593 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.601 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-11T22:38:42.602 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.610 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-11T22:38:42.610 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.618 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-11T22:38:42.618 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.627 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-11T22:38:42.627 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.635 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:169157fa:::existing_81:head# 2020-03-11T22:38:42.635 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.643 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:17731382:::existing_58:head# 2020-03-11T22:38:42.643 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.652 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-11T22:38:42.652 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.660 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-11T22:38:42.660 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.668 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-11T22:38:42.669 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.677 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-11T22:38:42.677 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.685 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-11T22:38:42.685 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.721 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:28566edb:::existing_78:head# 2020-03-11T22:38:42.722 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.733 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:2a101182:::existing_20:head# 2020-03-11T22:38:42.733 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.741 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-11T22:38:42.741 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.749 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-11T22:38:42.749 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.758 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-11T22:38:42.758 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.766 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-11T22:38:42.766 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.774 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-11T22:38:42.774 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.783 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-11T22:38:42.783 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.791 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-11T22:38:42.791 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.799 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-11T22:38:42.800 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.808 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-11T22:38:42.808 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.842 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-11T22:38:42.842 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.858 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:53588912:::existing_9:head# 2020-03-11T22:38:42.858 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.866 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-11T22:38:42.866 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.913 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-11T22:38:42.914 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.918 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-11T22:38:42.918 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.927 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-11T22:38:42.927 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.935 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:602f83fe:::foo:head# 2020-03-11T22:38:42.935 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.943 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-11T22:38:42.944 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.952 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:628918ca:::existing_79:head# 2020-03-11T22:38:42.952 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.960 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-11T22:38:42.960 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.969 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-11T22:38:42.969 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.977 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-11T22:38:42.977 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.985 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-11T22:38:42.985 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:42.993 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-11T22:38:42.994 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.002 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-11T22:38:43.002 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.010 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-11T22:38:43.011 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.047 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-11T22:38:43.047 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.052 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:750ace21:::existing_19:head# 2020-03-11T22:38:43.052 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.060 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-11T22:38:43.061 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.162 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:79881152:::existing_63:head# 2020-03-11T22:38:43.162 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.177 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-11T22:38:43.177 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.186 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:81490677:::existing_76:head# 2020-03-11T22:38:43.186 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.194 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-11T22:38:43.194 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.202 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-11T22:38:43.202 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.211 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:38:43.211 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.219 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-11T22:38:43.219 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.227 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:92731e71:::existing_89:head# 2020-03-11T22:38:43.228 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.236 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:928205cf:::existing_82:head# 2020-03-11T22:38:43.236 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.244 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:93be2267:::existing_24:head# 2020-03-11T22:38:43.244 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.252 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-11T22:38:43.253 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.297 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-11T22:38:43.298 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.303 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-11T22:38:43.303 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.311 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-11T22:38:43.311 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.320 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-11T22:38:43.320 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.330 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:a504a533:::existing_90:head# 2020-03-11T22:38:43.330 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.336 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:a691337e:::existing_61:head# 2020-03-11T22:38:43.336 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.344 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:38:43.345 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.353 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-11T22:38:43.353 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.383 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-11T22:38:43.383 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.395 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-11T22:38:43.395 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.440 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-11T22:38:43.440 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.473 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:b343e277:::existing_88:head# 2020-03-11T22:38:43.473 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.478 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-11T22:38:43.478 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.510 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-11T22:38:43.511 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.512 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-11T22:38:43.512 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.520 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-11T22:38:43.520 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.528 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:bebec877:::existing_74:head# 2020-03-11T22:38:43.528 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.536 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-11T22:38:43.536 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.545 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:c156033e:::existing_100:head# 2020-03-11T22:38:43.545 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.553 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-11T22:38:43.553 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.561 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:38:43.562 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.570 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:38:43.570 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.578 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-11T22:38:43.578 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.587 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-11T22:38:43.587 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.595 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-11T22:38:43.595 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.603 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-11T22:38:43.603 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.612 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-11T22:38:43.612 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.662 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-11T22:38:43.662 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.678 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:38:43.679 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.687 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-11T22:38:43.687 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.695 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-11T22:38:43.695 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.703 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-11T22:38:43.704 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.712 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-11T22:38:43.712 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.720 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:e255b710:::existing_55:head# 2020-03-11T22:38:43.720 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.728 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-11T22:38:43.729 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.737 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-11T22:38:43.737 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.745 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-11T22:38:43.746 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.754 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:f6809456:::existing_11:head# 2020-03-11T22:38:43.754 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.762 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-11T22:38:43.762 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.770 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-11T22:38:43.770 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.779 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-11T22:38:43.779 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.806 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-11T22:38:43.806 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:38:43.812 INFO:tasks.workunit.client.0.smithi016.stdout:write_pg epoch 45 info 1.0( v 45'102 lc 0'0 (29'90,45'102] local-lis/les=43/45 n=101 ec=23/23 lis/c=43/23 les/c/f=45/24/0 sis=43 pruub=7.632552175s) 2020-03-11T22:38:43.820 INFO:tasks.workunit.client.0.smithi016.stdout:Import successful 2020-03-11T22:38:44.341 INFO:tasks.workunit.client.0.smithi016.stdout:reviving divergent 1 2020-03-11T22:38:44.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-11T22:38:44.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:44.693 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:44.693 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:38:32.568050+0000 39'101 45:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:38:44.693 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:38:44.694 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:44.695 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:38:44.706 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:38:44.706 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:44.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:44.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:38:44.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:44.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:44.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:44.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:44.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:44.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:44.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:44.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:44.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:44.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:44.709 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:44.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:44.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:44.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:44.709 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:44.709 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:44.710 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:38:44.710 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:38:44.710 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:38:44.710 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:44.710 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:44.710 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:44.710 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:44.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:44.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:44.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:44.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:44.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:44.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:38:44.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:44.711 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:38:44.712 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:38:44.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:38:44.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:44.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:44.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:44.734 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:44.731+0000 7f458484cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:44.735 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:44.733+0000 7f458484cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:44.737 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:44.735+0000 7f458484cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:45.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:38:45.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:45.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:45.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:45.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:45.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:45.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:45.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:45.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:45.072 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:38:45.255 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:45.252+0000 7f458484cec0 -1 Falling back to public interface 2020-03-11T22:38:45.439 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/33049,v1:127.0.0.1:6819/33049] [v2:127.0.0.1:6820/33049,v1:127.0.0.1:6821/33049] exists,up f4026fd4-5a11-42e2-b606-f350d010eed0 2020-03-11T22:38:45.439 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:38:45.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:45.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:45.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:45.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:38:45.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:45.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:45.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:45.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:45.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:45.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:45.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:45.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:45.775 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:38:45.773+0000 7f458484cec0 -1 osd.1 45 log_to_monitors {default=true} 2020-03-11T22:38:45.803 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/33049,v1:127.0.0.1:6819/33049] [v2:127.0.0.1:6820/33049,v1:127.0.0.1:6821/33049] exists,up f4026fd4-5a11-42e2-b606-f350d010eed0 2020-03-11T22:38:45.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:45.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:45.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:45.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-11T22:39:05.804 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:39:05.805 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:39:05.805 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:39:05.806 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:05.806 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:05.806 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:05.806 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.1.asok 2020-03-11T22:39:05.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:39:05.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:39:05.908 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:39:05.908 INFO:tasks.workunit.client.0.smithi016.stdout: "ops": [], 2020-03-11T22:39:05.908 INFO:tasks.workunit.client.0.smithi016.stdout: "num_ops": 0 2020-03-11T22:39:05.908 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:39:05.916 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-11T22:39:05.916 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:05.916 INFO:tasks.workunit.client.0.smithi016.stdout:allowing recovery 2020-03-11T22:39:06.257 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:06.257 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:38:47.185082+0000 45'102 48:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:39:06.258 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:39:06.258 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:06.259 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:39:06.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:39:06.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:39:06.396 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:06.403 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:39:06.403 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:39:06.531 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:06.538 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:39:06.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:39:06.666 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:06.674 INFO:tasks.workunit.client.0.smithi016.stdout:reading divergent objects 2020-03-11T22:39:06.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-11T22:39:06.675 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:07.014 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:07.014 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:38:47.185082+0000 45'102 48:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:28.688212+0000 0'0 2020-03-11T22:36:28.688212+0000 0 2020-03-11T22:39:07.014 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:39:07.015 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:07.016 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:39:07.024 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-11T22:39:07.025 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-11T22:39:07.026 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:07.026 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:39:07.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:07.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:39:07.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:07.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:39:07.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:07.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:39:07.143 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:39:07.172 INFO:tasks.workunit.client.0.smithi016.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:39:07.201 INFO:tasks.workunit.client.0.smithi016.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:39:07.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-11T22:39:07.203 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2020-03-11T22:36:53.266153+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: keeping 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2020-03-11T22:36:53.266153+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'106 last_divergent_update: 29'106 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4652.0:1 2020-03-11T22:37:48.243958+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4534.0:1 2020-03-11T22:36:53.258644+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4534.0:1 2020-03-11T22:36:53.258644+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2020-03-11T22:36:53.264281+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2020-03-11T22:36:53.264281+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2020-03-11T22:36:53.264281+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'104 (29'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2020-03-11T22:36:53.262797+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: keeping 29'104 (29'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2020-03-11T22:36:53.262797+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'104 (29'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2020-03-11T22:36:53.262797+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4528.0:1 2020-03-11T22:36:53.244472+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: keeping 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4528.0:1 2020-03-11T22:36:53.244472+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'101 last_divergent_update: 29'101 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4528.0:1 2020-03-11T22:36:53.244472+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-03-11T22:36:53.253389+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 20 _merge_object_divergent_entries: keeping 29'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-03-11T22:36:53.253389+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'102 last_divergent_update: 29'102 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:32.480+0000 7f3da298f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.17411.out 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi016.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17411.out': No such file or directory 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi016.stdout:success 2020-03-11T22:39:07.703 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' does not exist 2020-03-11T22:39:07.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-11T22:39:07.712 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:07.712 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:07.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:07.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:07.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:07.820 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:07.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:39:07.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:39:07.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:39:07.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:39:07.821 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:07.821 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:07.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:07.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:07.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:07.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:07.826 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:39:07.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:39:07.827 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:39:07.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:39:07.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:39:07.828 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:39:07.829 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:07.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:39:07.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:39:07.830 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:07.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:39:07.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:39:07.832 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:39:07.844 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:39:07.844 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:07.844 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:07.844 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:39:07.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:39:07.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:39:07.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:39:07.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:39:07.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:39:07.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:39:07.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:39:07.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:39:07.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:39:07.847 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:07.847 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:07.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:07.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:07.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:07.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:07.849 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:39:07.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:39:07.849 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:39:07.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:39:07.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:39:07.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:39:07.851 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:07.851 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:39:07.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:39:07.852 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:07.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:39:07.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:39:07.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:39:07.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:39:07.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:07.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:07.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:39:07.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:39:07.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:39:07.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:39:07.857 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:39:07.857 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:07.857 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:07.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17411 2020-03-11T22:39:07.859 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:39:07.859 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:39:07.859 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:39:07.859 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:39:07.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-11T22:39:07.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-11T22:39:07.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-11T22:39:07.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-11T22:39:07.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-11T22:39:07.860 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-11T22:39:07.862 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-11T22:39:07.862 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-11T22:39:07.862 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:39:07.862 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:39:07.862 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-11T22:39:07.862 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:39:07.862 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:39:07.863 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:39:07.863 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:39:07.863 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:39:07.863 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:39:08.162 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:39:08.162 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:08.162 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:08.162 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:08.162 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:08.162 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:08.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:39:08.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:39:08.354 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:39:08.356 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:39:08.356 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:39:08.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:39:08.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:39:08.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:39:08.357 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:39:08.357 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:39:08.358 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:39:08.358 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:08.358 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:08.358 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:08.358 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-mon.a.asok 2020-03-11T22:39:08.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:39:08.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17411/ceph-mon.a.asok config get fsid 2020-03-11T22:39:08.453 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:39:08.453 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:39:08.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:39:08.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:39:08.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:39:08.454 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:39:08.455 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:39:08.455 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:39:08.455 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:08.455 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:08.455 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:08.456 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-mon.a.asok 2020-03-11T22:39:08.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:39:08.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17411/ceph-mon.a.asok config get mon_host 2020-03-11T22:39:08.542 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-11T22:39:08.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:39:08.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:39:08.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:39:08.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:39:08.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:39:08.544 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:39:09.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:39:09.441 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:09.441 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:09.441 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:09.441 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:09.441 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:09.441 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:39:09.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:39:09.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:39:09.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:09.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-11T22:39:09.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:09.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:09.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:39:09.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:09.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:39:09.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:09.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:09.473 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:09.473 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:09.473 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:09.473 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:09.473 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:09.473 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:09.474 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:39:09.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:39:09.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:09.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:09.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:09.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:09.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:09.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:09.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:09.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:09.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:09.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:09.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:39:09.476 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:09.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1adeefef-8ac6-40d7-ba02-a2dc9602cac5 2020-03-11T22:39:09.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 1adeefef-8ac6-40d7-ba02-a2dc9602cac5' 2020-03-11T22:39:09.477 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:09.477 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 1adeefef-8ac6-40d7-ba02-a2dc9602cac5 2020-03-11T22:39:09.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCNaGles3gYHRAAydiq9kYuaeNvlkzURJUUTA== 2020-03-11T22:39:09.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCNaGles3gYHRAAydiq9kYuaeNvlkzURJUUTA=="}' 2020-03-11T22:39:09.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1adeefef-8ac6-40d7-ba02-a2dc9602cac5 -i td/divergent-priors/0/new.json 2020-03-11T22:39:09.848 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:39:09.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:39:09.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCNaGles3gYHRAAydiq9kYuaeNvlkzURJUUTA== --osd-uuid 1adeefef-8ac6-40d7-ba02-a2dc9602cac5 2020-03-11T22:39:09.888 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:09.885+0000 7f0e1cc75ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:09.890 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:09.887+0000 7f0e1cc75ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:09.891 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:09.888+0000 7f0e1cc75ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:09.891 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:09.888+0000 7f0e1cc75ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:09.892 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:09.888+0000 7f0e1cc75ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:09.892 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:09.888+0000 7f0e1cc75ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:09.929 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:09.925+0000 7f0e1cc75ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:39:13.012 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:39:13.012 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:13.013 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:39:13.013 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:39:13.013 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:13.511 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:39:13.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:39:13.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:13.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:13.512 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:13.512 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:13.538 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:13.535+0000 7f49648e5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:13.540 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:13.537+0000 7f49648e5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:13.542 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:13.539+0000 7f49648e5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:13.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:39:13.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:13.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:39:13.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:13.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:13.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:13.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:13.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:13.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:13.899 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:39:14.058 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:14.054+0000 7f49648e5ec0 -1 Falling back to public interface 2020-03-11T22:39:14.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:14.778 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:14.775+0000 7f49648e5ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:39:15.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:15.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:15.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:15.290 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:39:15.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:15.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:15.729 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34544,v1:127.0.0.1:6803/34544] [v2:127.0.0.1:6804/34544,v1:127.0.0.1:6805/34544] exists,up 1adeefef-8ac6-40d7-ba02-a2dc9602cac5 2020-03-11T22:39:15.729 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 55b263f4-40f2-4800-a15b-74333e0f0c12 2020-03-11T22:39:15.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:15.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-11T22:39:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:39:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:39:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:39:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:39:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:15.732 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:15.732 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:15.732 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:15.733 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:39:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:39:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:39:15.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=55b263f4-40f2-4800-a15b-74333e0f0c12 2020-03-11T22:39:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 55b263f4-40f2-4800-a15b-74333e0f0c12' 2020-03-11T22:39:15.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:15.747 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCTaGlensdpLBAA2Inab5yb1tcFXmCXbbjm9A== 2020-03-11T22:39:15.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCTaGlensdpLBAA2Inab5yb1tcFXmCXbbjm9A=="}' 2020-03-11T22:39:15.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 55b263f4-40f2-4800-a15b-74333e0f0c12 -i td/divergent-priors/1/new.json 2020-03-11T22:39:16.268 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:39:16.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:39:16.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCTaGlensdpLBAA2Inab5yb1tcFXmCXbbjm9A== --osd-uuid 55b263f4-40f2-4800-a15b-74333e0f0c12 2020-03-11T22:39:16.309 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:16.306+0000 7efda7fcbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:16.311 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:16.308+0000 7efda7fcbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:16.312 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:16.309+0000 7efda7fcbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:16.312 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:16.309+0000 7efda7fcbec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:16.312 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:16.309+0000 7efda7fcbec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:16.313 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:16.309+0000 7efda7fcbec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:16.361 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:16.358+0000 7efda7fcbec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:39:19.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:39:19.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:19.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:39:19.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:19.429 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:39:19.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:39:19.928 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:19.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:19.957 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:19.953+0000 7ffb95f1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:19.958 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:19.955+0000 7ffb95f1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:19.959 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:19.956+0000 7ffb95f1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:20.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:39:20.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:20.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:39:20.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:20.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:20.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:20.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:20.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:20.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:20.291 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:39:20.477 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:20.474+0000 7ffb95f1fec0 -1 Falling back to public interface 2020-03-11T22:39:20.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:21.277 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:21.273+0000 7ffb95f1fec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:39:21.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:21.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:21.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:21.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:21.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:21.658 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:39:22.038 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34899,v1:127.0.0.1:6811/34899] [v2:127.0.0.1:6812/34899,v1:127.0.0.1:6813/34899] exists,up 55b263f4-40f2-4800-a15b-74333e0f0c12 2020-03-11T22:39:22.038 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:22.038 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:22.038 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:22.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:22.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-11T22:39:22.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:22.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:22.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:39:22.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:22.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:39:22.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:22.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:22.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:22.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:22.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:39:22.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:39:22.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:22.041 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:22.041 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:22.041 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:22.041 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:22.041 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:22.041 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:22.042 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:22.042 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:22.042 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:39:22.042 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:39:22.042 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:22.042 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:22.042 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:22.042 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:22.043 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:22.043 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:22.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:22.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:22.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:22.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:22.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:39:22.045 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:22.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=50c76265-8eaa-44a7-8947-296ee0f56e9d 2020-03-11T22:39:22.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 50c76265-8eaa-44a7-8947-296ee0f56e9d' 2020-03-11T22:39:22.046 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:22.046 INFO:tasks.workunit.client.0.smithi016.stdout:add osd2 50c76265-8eaa-44a7-8947-296ee0f56e9d 2020-03-11T22:39:22.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCaaGleIVFMAxAAHNcKsPszGTNc2819o/xDeQ== 2020-03-11T22:39:22.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCaaGleIVFMAxAAHNcKsPszGTNc2819o/xDeQ=="}' 2020-03-11T22:39:22.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 50c76265-8eaa-44a7-8947-296ee0f56e9d -i td/divergent-priors/2/new.json 2020-03-11T22:39:22.498 INFO:tasks.workunit.client.0.smithi016.stdout:2 2020-03-11T22:39:22.507 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:39:22.508 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCaaGleIVFMAxAAHNcKsPszGTNc2819o/xDeQ== --osd-uuid 50c76265-8eaa-44a7-8947-296ee0f56e9d 2020-03-11T22:39:22.538 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:22.535+0000 7f1d785c8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.550 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:22.547+0000 7f1d785c8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.551 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:22.548+0000 7f1d785c8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.552 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:22.549+0000 7f1d785c8ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:22.552 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:22.549+0000 7f1d785c8ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:22.552 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:22.549+0000 7f1d785c8ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:22.617 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:22.614+0000 7f1d785c8ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:39:26.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:39:26.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:26.119 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd2 key to auth repository 2020-03-11T22:39:26.120 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:39:26.120 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:26.602 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:39:26.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:26.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:26.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:26.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:26.603 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:39:26.630 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:26.627+0000 7fc5c185fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:26.632 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:26.629+0000 7fc5c185fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:26.634 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:26.630+0000 7fc5c185fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:27.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:39:27.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:27.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:39:27.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:27.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:27.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:27.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:27.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:27.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:27.055 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:39:27.150 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:27.146+0000 7fc5c185fec0 -1 Falling back to public interface 2020-03-11T22:39:27.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:27.901 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:39:27.888+0000 7fc5c185fec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:39:28.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:28.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:28.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:28.435 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:39:28.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:28.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:28.815 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35254,v1:127.0.0.1:6819/35254] [v2:127.0.0.1:6820/35254,v1:127.0.0.1:6821/35254] exists,up 50c76265-8eaa-44a7-8947-296ee0f56e9d 2020-03-11T22:39:28.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:28.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:28.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:28.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-11T22:39:29.561 INFO:tasks.workunit.client.0.smithi016.stderr:noout is set 2020-03-11T22:39:29.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-11T22:39:30.510 INFO:tasks.workunit.client.0.smithi016.stderr:noin is set 2020-03-11T22:39:30.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-11T22:39:31.423 INFO:tasks.workunit.client.0.smithi016.stderr:nodown is set 2020-03-11T22:39:31.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-11T22:39:31.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:39:31.927 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' already exists 2020-03-11T22:39:31.935 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:39:32.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-11T22:39:33.769 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 size to 3 2020-03-11T22:39:33.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-11T22:39:34.428 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 min_size to 2 2020-03-11T22:39:34.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-11T22:39:35.089 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 pg_autoscale_mode to on 2020-03-11T22:39:35.097 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-11T22:39:35.097 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:35.098 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:35.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:35.475 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:39:35.475 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:39:35.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:35.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:35.476 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:35.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:39:35.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:39:35.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:35.612 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:35.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:39:35.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2020-03-11T22:39:35.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:35.750 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:35.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-03-11T22:39:35.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-60129542148' 2020-03-11T22:39:35.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:35.887 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:39:35.887 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:35.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:35.888 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:39:35.889 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:35.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:39:35.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:39:35.890 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:35.891 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:39:36.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836485 -lt 21474836487 2020-03-11T22:39:36.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:37.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:37.251 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:37.624 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836487 2020-03-11T22:39:37.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:37.625 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:39:37.626 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:37.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:37.626 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:39:37.626 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:37.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:39:37.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:39:37.628 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:39:37.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:37.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:39:37.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:37.992 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542148 2020-03-11T22:39:37.992 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:37.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:37.993 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542148 2020-03-11T22:39:37.993 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:37.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542148 2020-03-11T22:39:37.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-03-11T22:39:37.994 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 60129542148 2020-03-11T22:39:37.997 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:38.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542148 2020-03-11T22:39:38.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-11T22:39:38.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:38.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:38.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:38.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:38.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:39:38.366 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:38.366 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:38.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:38.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:38.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:38.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:38.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:38.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:38.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:38.463 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:38.827 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:38.827 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:39:38.828 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:39:38.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:38.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:38.828 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:39:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:39:38.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:38.964 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:39.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:39:39.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2020-03-11T22:39:39.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:39.109 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:39.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542151 2020-03-11T22:39:39.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-60129542151' 2020-03-11T22:39:39.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:39.245 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:39:39.245 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:39.246 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:39.246 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:39:39.246 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:39.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:39:39.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:39:39.254 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:39.255 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:39:39.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:39:39.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:40.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:40.613 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:40.986 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:39:40.987 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:41.988 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:39:41.988 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:42.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:39:42.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:42.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:39:42.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:42.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:42.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:39:42.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:42.368 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:39:42.368 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:39:42.369 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:42.369 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:39:42.752 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:39:42.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:42.753 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542151 2020-03-11T22:39:42.754 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:42.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:42.754 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542151 2020-03-11T22:39:42.754 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:42.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542151 2020-03-11T22:39:42.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2020-03-11T22:39:42.755 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:42.756 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 60129542151 2020-03-11T22:39:43.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542151 -lt 60129542151 2020-03-11T22:39:43.154 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:43.154 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:43.154 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:44.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:39:44.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:44.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:44.009 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:44.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:39:44.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:39:44.483 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:39:44.483 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-11T22:39:44.483 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-11T22:39:44.843 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:39:44.851 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-11T22:39:44.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-11T22:39:44.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:44.852 INFO:tasks.workunit.client.0.smithi016.stdout:primary and soon to be divergent is 1 2020-03-11T22:39:45.193 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:45.194 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:39:33.076644+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:39:45.194 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:39:45.194 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:45.195 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:39:45.203 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-11T22:39:45.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:45.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-11T22:39:45.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-11T22:39:45.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:45.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-11T22:39:45.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-11T22:39:45.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:45.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-11T22:39:45.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-11T22:39:45.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-11T22:39:45.205 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-11T22:39:45.205 INFO:tasks.workunit.client.0.smithi016.stdout:writing initial objects 2020-03-11T22:39:45.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-11T22:39:45.258 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.258 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-11T22:39:45.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-11T22:39:45.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-11T22:39:45.478 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.479 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-11T22:39:45.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-11T22:39:45.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-11T22:39:45.675 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.675 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-11T22:39:45.715 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.715 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-11T22:39:45.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-11T22:39:45.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-11T22:39:45.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-11T22:39:45.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.891 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-11T22:39:45.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.934 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-11T22:39:45.987 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.987 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-11T22:39:46.079 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.079 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-11T22:39:46.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-11T22:39:46.217 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-11T22:39:46.257 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.257 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-11T22:39:46.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-11T22:39:46.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-11T22:39:46.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-11T22:39:46.492 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.492 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-11T22:39:46.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-11T22:39:46.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-11T22:39:46.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-11T22:39:46.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-11T22:39:46.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-11T22:39:46.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-11T22:39:46.940 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.941 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-11T22:39:46.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-11T22:39:47.098 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.098 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-11T22:39:47.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-11T22:39:47.225 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-11T22:39:47.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-11T22:39:47.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-11T22:39:47.349 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.349 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-11T22:39:47.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-11T22:39:47.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-11T22:39:47.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-11T22:39:47.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-11T22:39:47.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-11T22:39:47.791 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.791 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-11T22:39:47.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-11T22:39:47.907 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.907 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-11T22:39:47.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-11T22:39:47.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-11T22:39:48.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-11T22:39:48.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.100 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-11T22:39:48.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-11T22:39:48.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.225 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-11T22:39:48.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-11T22:39:48.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-11T22:39:48.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-11T22:39:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-11T22:39:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-11T22:39:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-11T22:39:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-11T22:39:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-11T22:39:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-11T22:39:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-11T22:39:49.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-11T22:39:49.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-11T22:39:49.246 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.246 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-11T22:39:49.293 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.293 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-11T22:39:49.377 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.377 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-11T22:39:49.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-11T22:39:49.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-11T22:39:49.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-11T22:39:49.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-11T22:39:49.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-11T22:39:49.725 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.725 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-11T22:39:49.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-11T22:39:49.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-11T22:39:49.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-11T22:39:50.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-11T22:39:50.068 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.068 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-11T22:39:50.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-11T22:39:50.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-11T22:39:50.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-11T22:39:50.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-11T22:39:50.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-11T22:39:50.401 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-11T22:39:50.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-11T22:39:50.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-11T22:39:50.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.596 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-11T22:39:50.726 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-11T22:39:50.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-11T22:39:50.891 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-11T22:39:50.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-11T22:39:51.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-11T22:39:51.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-11T22:39:51.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-11T22:39:51.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-11T22:39:51.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-11T22:39:51.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-11T22:39:51.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-11T22:39:51.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-11T22:39:51.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-11T22:39:51.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:39:51.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-11T22:39:51.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:51.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:51.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:51.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:51.575 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:39:51.576 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:51.576 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:51.576 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:51.576 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:51.576 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:51.633 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:51.634 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:51.634 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:51.634 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:51.634 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:52.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:52.007 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:39:52.007 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:39:52.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:52.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:52.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:52.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836495 2020-03-11T22:39:52.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836495' 2020-03-11T22:39:52.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:52.144 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:52.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:39:52.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836495 1-42949672972' 2020-03-11T22:39:52.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:52.281 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:52.416 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542155 2020-03-11T22:39:52.416 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836495 1-42949672972 2-60129542155' 2020-03-11T22:39:52.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:52.417 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836495 2020-03-11T22:39:52.417 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:52.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:52.418 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836495 2020-03-11T22:39:52.418 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:52.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836495 2020-03-11T22:39:52.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-03-11T22:39:52.420 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:52.420 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836495 2020-03-11T22:39:52.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836495 2020-03-11T22:39:52.794 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:53.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:53.795 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:54.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836495 -lt 21474836495 2020-03-11T22:39:54.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:54.171 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:39:54.171 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:54.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:54.172 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:39:54.172 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:54.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:39:54.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:39:54.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:54.173 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:39:54.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672972 2020-03-11T22:39:54.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:54.560 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542155 2020-03-11T22:39:54.560 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:54.560 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:54.561 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542155 2020-03-11T22:39:54.561 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:54.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542155 2020-03-11T22:39:54.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2020-03-11T22:39:54.562 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:54.563 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 60129542155 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542155 -lt 60129542155 2020-03-11T22:39:54.962 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:54.963 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:55.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:55.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:55.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:55.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:55.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:55.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:55.789 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:39:55.789 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:55.789 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:55.790 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:56.262 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:39:56.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:39:56.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:39:56.263 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-11T22:39:56.264 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:39:56.264 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:39:56.264 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:39:56.264 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:39:56.633 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-11T22:39:56.633 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-11T22:39:56.634 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:56.634 INFO:tasks.workunit.client.0.smithi016.stdout:blackholing osds 0 2 2020-03-11T22:39:56.995 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:56.995 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:39:33.076644+0000 30'100 30:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:39:56.995 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:39:56.995 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:56.996 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:39:57.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:57.005 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:39:57.006 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:39:57.006 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:39:57.006 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:57.006 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:57.006 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:57.006 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.0.asok 2020-03-11T22:39:57.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:39:57.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:39:57.097 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:39:57.097 INFO:tasks.workunit.client.0.smithi016.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:39:57.097 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:39:57.104 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:57.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:39:57.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:39:57.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:39:57.106 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:57.106 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:57.107 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:39:57.107 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.2.asok 2020-03-11T22:39:57.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:39:57.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:39:57.197 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:39:57.197 INFO:tasks.workunit.client.0.smithi016.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:39:57.197 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:39:57.204 INFO:tasks.workunit.client.0.smithi016.stdout:create new divergent objects 2020-03-11T22:39:57.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-11T22:39:57.204 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-11T22:39:57.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:57.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:57.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-11T22:39:57.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-11T22:39:57.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-11T22:39:57.206 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-11T22:39:57.207 INFO:tasks.workunit.client.0.smithi016.stdout:writing divergent objects 2020-03-11T22:39:57.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:39:57.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:39:57.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:39:57.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-11T22:39:57.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:39:57.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-11T22:39:57.209 INFO:tasks.workunit.client.0.smithi016.stdout:remove divergent objects 2020-03-11T22:39:57.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:39:57.210 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-11T22:39:57.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-11T22:39:57.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:39:57.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-11T22:39:57.211 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-11T22:39:57.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-11T22:39:57.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-11T22:39:57.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-11T22:40:07.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-11T22:40:07.215 INFO:tasks.workunit.client.0.smithi016.stderr:rados: no process found 2020-03-11T22:40:07.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-11T22:40:07.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:07.216 INFO:tasks.workunit.client.0.smithi016.stdout:killing all the osds 2020-03-11T22:40:07.570 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:07.571 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:39:33.076644+0000 30'106 30:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:07.571 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:40:07.571 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:07.572 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:40:07.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:40:07.581 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:07.581 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:07.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:07.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:07.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:07.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:07.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:40:07.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-11T22:40:08.246 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already down. 2020-03-11T22:40:08.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:40:08.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-11T22:40:08.794 INFO:tasks.workunit.client.0.smithi016.stderr:osd.1 is already down. 2020-03-11T22:40:08.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:40:08.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-11T22:40:09.346 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already down. 2020-03-11T22:40:09.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:09.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-11T22:40:09.973 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already out. 2020-03-11T22:40:09.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:09.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-11T22:40:10.523 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already out. 2020-03-11T22:40:10.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-11T22:40:10.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:10.532 INFO:tasks.workunit.client.0.smithi016.stdout:bringing up non_divergent 0 2 2020-03-11T22:40:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:39:33.076644+0000 30'106 30:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:10.870 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:40:10.871 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:10.872 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:40:10.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:10.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-11T22:40:10.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:10.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:10.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:40:10.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:10.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:40:10.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:10.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:10.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:10.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:10.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:40:10.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:40:10.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:10.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:10.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:10.885 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:10.885 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:10.885 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:10.886 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:10.886 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:10.886 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:40:10.886 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:40:10.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:40:10.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:10.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:10.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:10.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:10.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:10.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:10.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:10.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:10.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:40:10.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:40:10.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:10.888 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:40:10.888 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:40:10.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:40:10.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:10.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:10.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:10.910 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:10.907+0000 7f87d4c34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:10.912 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:10.909+0000 7f87d4c34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:10.913 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:10.910+0000 7f87d4c34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:11.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:40:11.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:11.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:40:11.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:11.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:11.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:11.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:11.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:11.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:11.251 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:40:11.429 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:11.425+0000 7f87d4c34ec0 -1 Falling back to public interface 2020-03-11T22:40:11.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:12.254 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:12.251+0000 7f87d4c34ec0 -1 osd.0 30 log_to_monitors {default=true} 2020-03-11T22:40:12.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:12.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:12.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:12.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:12.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:12.614 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:40:12.993 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/39346,v1:127.0.0.1:6803/39346] [v2:127.0.0.1:6804/39346,v1:127.0.0.1:6805/39346] exists,up 1adeefef-8ac6-40d7-ba02-a2dc9602cac5 2020-03-11T22:40:12.993 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:40:12.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:12.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:12.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:12.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:12.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-11T22:40:12.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:12.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:12.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:40:12.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:12.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:40:12.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:12.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:12.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:12.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:12.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:40:12.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:40:12.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:12.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:12.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:12.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:12.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:12.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:12.996 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:12.997 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:12.997 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:40:12.997 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:40:12.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:40:12.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:12.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:12.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:12.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:12.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:12.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:12.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:12.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:12.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:40:12.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:40:12.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:12.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:40:12.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:40:12.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:12.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:12.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:13.019 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:13.016+0000 7f4ec7e24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:13.033 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:13.030+0000 7f4ec7e24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:13.035 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:13.032+0000 7f4ec7e24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:13.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:40:13.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:13.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:13.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:13.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:13.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:13.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:13.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:13.364 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:13.364 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:40:13.551 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:13.547+0000 7f4ec7e24ec0 -1 Falling back to public interface 2020-03-11T22:40:13.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:14.480 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:40:14.477+0000 7f4ec7e24ec0 -1 osd.2 30 log_to_monitors {default=true} 2020-03-11T22:40:14.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:14.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:14.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:14.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:14.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:14.780 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:40:15.159 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 33 last_clean_interval [14,30) [v2:127.0.0.1:6810/39540,v1:127.0.0.1:6811/39540] [v2:127.0.0.1:6812/39540,v1:127.0.0.1:6813/39540] exists,up 50c76265-8eaa-44a7-8947-296ee0f56e9d 2020-03-11T22:40:15.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:15.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:15.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:15.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:15.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-11T22:40:15.765 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already in. 2020-03-11T22:40:15.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:15.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-11T22:40:16.604 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already in. 2020-03-11T22:40:16.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:16.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-11T22:40:16.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:16.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:16.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:16.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:16.613 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:16.614 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:16.614 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:16.614 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:16.614 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:16.614 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:16.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:16.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:16.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:16.675 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:16.675 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:17.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:17.054 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:40:17.054 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:40:17.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:17.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:17.055 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:17.190 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:40:17.190 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659' 2020-03-11T22:40:17.190 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:17.191 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:17.286 INFO:tasks.workunit.client.0.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:40:17.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:40:17.295 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659 1-' 2020-03-11T22:40:17.295 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:17.295 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:17.431 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=163208757251 2020-03-11T22:40:17.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659 1- 2-163208757251' 2020-03-11T22:40:17.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:17.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822659 2020-03-11T22:40:17.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:17.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:17.433 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822659 2020-03-11T22:40:17.433 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:17.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:40:17.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822659' 2020-03-11T22:40:17.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:17.435 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 154618822659 2020-03-11T22:40:17.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 154618822659 2020-03-11T22:40:17.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:18.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:18.799 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:19.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822660 -lt 154618822659 2020-03-11T22:40:19.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:19.225 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:40:19.225 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:19.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:19.226 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:40:19.226 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:19.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:40:19.228 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:40:19.228 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:19.228 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 2020-03-11T22:40:19.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 2020-03-11T22:40:19.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672976: unary operator expected 2020-03-11T22:40:19.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:19.594 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-163208757251 2020-03-11T22:40:19.594 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:19.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:19.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-163208757251 2020-03-11T22:40:19.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:19.596 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=163208757251 2020-03-11T22:40:19.596 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2020-03-11T22:40:19.596 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:19.596 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 163208757251 2020-03-11T22:40:19.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 163208757251 -lt 163208757251 2020-03-11T22:40:19.960 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:19.961 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:19.961 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:20.431 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:40:20.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:20.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:20.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:20.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:20.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:20.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:20.433 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:20.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:20.787 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:20.788 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:20.788 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:21.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:21.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:40:21.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:40:21.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:40:21.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:21.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:40:21.356 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:21.357 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:21.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:21.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:21.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:21.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:21.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:21.358 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:21.710 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:21.711 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:21.711 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:21.711 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:22.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:22.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:22.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:22.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:22.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:22.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:22.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:22.182 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:22.183 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:22.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:22.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:22.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:40:22.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:22.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:40:22.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:22.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:22.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:22.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:22.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:22.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:22.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:22.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:23.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:23.212 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:23.213 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:23.213 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:23.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:23.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:23.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:23.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:23.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:23.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:23.687 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:23.687 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:23.687 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:24.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:24.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:24.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:40:24.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:24.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:40:24.556 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:24.556 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:24.557 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:24.557 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:24.558 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:24.558 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:24.558 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:24.558 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:24.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:24.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:24.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:24.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:25.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:25.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:25.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:25.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:25.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:25.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:25.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:25.414 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:25.414 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:25.891 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:25.891 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:25.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:40:25.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:25.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:40:26.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:26.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:26.695 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:26.696 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:26.696 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:26.697 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:26.697 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:26.697 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:27.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:27.060 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:27.060 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:27.060 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:27.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:27.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:27.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:27.536 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:27.536 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:27.536 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:27.536 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:27.537 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:27.537 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:28.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:28.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:28.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:40:28.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:28.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:40:29.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:29.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:29.608 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:29.608 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:29.608 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:29.608 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:29.609 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:29.609 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:29.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:29.984 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:29.984 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:29.984 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:30.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:30.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:30.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:30.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:30.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:30.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:30.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:30.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:30.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:30.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:30.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:30.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:40:30.928 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:30.928 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:40:34.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:34.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:34.129 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:34.129 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:34.130 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:34.130 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:34.130 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:34.130 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:34.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:34.501 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:34.502 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:34.502 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:34.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:34.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:34.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:34.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:34.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:34.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:34.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:34.965 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:34.965 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:35.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:35.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:35.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:40:35.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:35.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:40:41.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:41.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:41.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:41.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:41.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:41.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:41.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:41.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:42.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:42.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:42.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:42.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:42.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:42.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:42.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:42.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:42.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:42.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:42.696 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:42.696 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:42.696 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:43.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:43.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:43.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:40:43.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:43.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:40:50.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:50.468 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:50.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:50.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:50.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:50.469 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:50.469 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:50.469 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:50.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:50.837 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:50.837 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:50.837 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:51.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:51.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:51.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:51.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:51.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:51.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:51.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:51.309 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:51.309 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:51.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:51.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:51.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:40:51.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:40:52.242 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:40:52.242 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_fingerprint": "3f106c27-51f9-4316-a5f5-e19ac3bff60f", 2020-03-11T22:40:52.243 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:40:52.243 INFO:tasks.workunit.client.0.smithi016.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:40:52.243 INFO:tasks.workunit.client.0.smithi016.stdout: "timestamp": "2020-03-11T22:40:52.239112+0000", 2020-03-11T22:40:52.243 INFO:tasks.workunit.client.0.smithi016.stdout: "tag": "", 2020-03-11T22:40:52.243 INFO:tasks.workunit.client.0.smithi016.stdout: "health": { 2020-03-11T22:40:52.243 INFO:tasks.workunit.client.0.smithi016.stdout: "status": "HEALTH_WARN", 2020-03-11T22:40:52.243 INFO:tasks.workunit.client.0.smithi016.stdout: "checks": { 2020-03-11T22:40:52.244 INFO:tasks.workunit.client.0.smithi016.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:40:52.244 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:52.244 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:40:52.244 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:40:52.244 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 17 2020-03-11T22:40:52.244 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.244 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [], 2020-03-11T22:40:52.244 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:40:52.245 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.245 INFO:tasks.workunit.client.0.smithi016.stdout: "OSD_DOWN": { 2020-03-11T22:40:52.245 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:52.245 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:40:52.245 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 osds down", 2020-03-11T22:40:52.245 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:40:52.245 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.246 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:40:52.246 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.246 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "osd.1 (root=default,host=smithi016) is down" 2020-03-11T22:40:52.246 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.246 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.246 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:40:52.246 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.247 INFO:tasks.workunit.client.0.smithi016.stdout: "PG_DEGRADED": { 2020-03-11T22:40:52.247 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:52.247 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:40:52.247 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-03-11T22:40:52.247 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 32 2020-03-11T22:40:52.247 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.247 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:40:52.247 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.248 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.248 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.248 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.248 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.248 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.248 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.248 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.248 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.249 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.249 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.249 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.249 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.249 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.249 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.250 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.250 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.250 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.250 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.250 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.250 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.251 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.251 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.251 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.251 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.251 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.251 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.251 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.252 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.252 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.252 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.252 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.252 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.252 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.252 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.252 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.253 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.253 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.253 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.253 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.253 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.253 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.253 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.254 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.254 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.254 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.254 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.254 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.254 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.254 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.254 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.255 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.255 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.255 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.255 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.255 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.255 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.255 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.256 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.256 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.256 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.256 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.256 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.256 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.256 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.256 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.257 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.257 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.257 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.257 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.257 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.257 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.257 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.258 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.258 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.258 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.258 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.258 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.258 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.258 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.258 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.259 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.259 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.259 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.259 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.259 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.259 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.259 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.260 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.260 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.260 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.260 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.260 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.260 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.260 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:52.261 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.261 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.261 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:40:52.261 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.261 INFO:tasks.workunit.client.0.smithi016.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:40:52.261 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:52.262 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:40:52.262 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:40:52.262 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:40:52.262 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.262 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:40:52.262 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.262 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:40:52.263 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.263 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.263 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:40:52.263 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.263 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.263 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:40:52.263 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.263 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.264 INFO:tasks.workunit.client.0.smithi016.stdout: "mutes": [] 2020-03-11T22:40:52.264 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.264 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_first_committed": 1, 2020-03-11T22:40:52.264 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_last_committed": 1, 2020-03-11T22:40:52.264 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap": { 2020-03-11T22:40:52.264 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:40:52.264 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:40:52.265 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:39:07.882909Z", 2020-03-11T22:40:52.265 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:39:07.882909Z", 2020-03-11T22:40:52.265 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release": 15, 2020-03-11T22:40:52.265 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:40:52.265 INFO:tasks.workunit.client.0.smithi016.stdout: "features": { 2020-03-11T22:40:52.265 INFO:tasks.workunit.client.0.smithi016.stdout: "persistent": [ 2020-03-11T22:40:52.265 INFO:tasks.workunit.client.0.smithi016.stdout: "kraken", 2020-03-11T22:40:52.266 INFO:tasks.workunit.client.0.smithi016.stdout: "luminous", 2020-03-11T22:40:52.266 INFO:tasks.workunit.client.0.smithi016.stdout: "mimic", 2020-03-11T22:40:52.266 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap-prune", 2020-03-11T22:40:52.266 INFO:tasks.workunit.client.0.smithi016.stdout: "nautilus", 2020-03-11T22:40:52.266 INFO:tasks.workunit.client.0.smithi016.stdout: "octopus" 2020-03-11T22:40:52.266 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.266 INFO:tasks.workunit.client.0.smithi016.stdout: "optional": [] 2020-03-11T22:40:52.266 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.267 INFO:tasks.workunit.client.0.smithi016.stdout: "mons": [ 2020-03-11T22:40:52.267 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.267 INFO:tasks.workunit.client.0.smithi016.stdout: "rank": 0, 2020-03-11T22:40:52.267 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "a", 2020-03-11T22:40:52.267 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:40:52.267 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.268 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.268 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.268 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:40:52.268 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 0 2020-03-11T22:40:52.268 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.268 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.268 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.269 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:40:52.269 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:40:52.269 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 0, 2020-03-11T22:40:52.269 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 0 2020-03-11T22:40:52.269 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.269 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.269 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.270 INFO:tasks.workunit.client.0.smithi016.stdout: "quorum": [ 2020-03-11T22:40:52.270 INFO:tasks.workunit.client.0.smithi016.stdout: 0 2020-03-11T22:40:52.270 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.270 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap": { 2020-03-11T22:40:52.270 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 44, 2020-03-11T22:40:52.270 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:40:52.270 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:39:08.641393+0000", 2020-03-11T22:40:52.270 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:40:16.442437+0000", 2020-03-11T22:40:52.271 INFO:tasks.workunit.client.0.smithi016.stdout: "last_up_change": "2020-03-11T22:40:14.553542+0000", 2020-03-11T22:40:52.271 INFO:tasks.workunit.client.0.smithi016.stdout: "last_in_change": "2020-03-11T22:40:16.221254+0000", 2020-03-11T22:40:52.271 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:40:52.271 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_num": 5800384, 2020-03-11T22:40:52.271 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_set": [ 2020-03-11T22:40:52.271 INFO:tasks.workunit.client.0.smithi016.stdout: "nodown", 2020-03-11T22:40:52.271 INFO:tasks.workunit.client.0.smithi016.stdout: "noin", 2020-03-11T22:40:52.272 INFO:tasks.workunit.client.0.smithi016.stdout: "noout", 2020-03-11T22:40:52.272 INFO:tasks.workunit.client.0.smithi016.stdout: "pglog_hardlimit", 2020-03-11T22:40:52.272 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snapdirs", 2020-03-11T22:40:52.272 INFO:tasks.workunit.client.0.smithi016.stdout: "recovery_deletes", 2020-03-11T22:40:52.272 INFO:tasks.workunit.client.0.smithi016.stdout: "sortbitwise" 2020-03-11T22:40:52.272 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.272 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_version": 7, 2020-03-11T22:40:52.272 INFO:tasks.workunit.client.0.smithi016.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:40:52.273 INFO:tasks.workunit.client.0.smithi016.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:40:52.273 INFO:tasks.workunit.client.0.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:40:52.273 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_snapshot": "", 2020-03-11T22:40:52.273 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_max": 1, 2020-03-11T22:40:52.273 INFO:tasks.workunit.client.0.smithi016.stdout: "max_osd": 3, 2020-03-11T22:40:52.273 INFO:tasks.workunit.client.0.smithi016.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:40:52.273 INFO:tasks.workunit.client.0.smithi016.stdout: "min_compat_client": "jewel", 2020-03-11T22:40:52.274 INFO:tasks.workunit.client.0.smithi016.stdout: "require_osd_release": "octopus", 2020-03-11T22:40:52.274 INFO:tasks.workunit.client.0.smithi016.stdout: "pools": [ 2020-03-11T22:40:52.274 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.274 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:40:52.274 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_name": "test", 2020-03-11T22:40:52.274 INFO:tasks.workunit.client.0.smithi016.stdout: "create_time": "2020-03-11T22:39:31.702645+0000", 2020-03-11T22:40:52.274 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": 1, 2020-03-11T22:40:52.274 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_names": "hashpspool", 2020-03-11T22:40:52.275 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:40:52.275 INFO:tasks.workunit.client.0.smithi016.stdout: "size": 3, 2020-03-11T22:40:52.275 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 2, 2020-03-11T22:40:52.275 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_rule": 0, 2020-03-11T22:40:52.275 INFO:tasks.workunit.client.0.smithi016.stdout: "object_hash": 2, 2020-03-11T22:40:52.275 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:40:52.275 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num": 32, 2020-03-11T22:40:52.276 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num": 1, 2020-03-11T22:40:52.276 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num_target": 32, 2020-03-11T22:40:52.276 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_target": 32, 2020-03-11T22:40:52.276 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_pending": 32, 2020-03-11T22:40:52.276 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pg_merge_meta": { 2020-03-11T22:40:52.276 INFO:tasks.workunit.client.0.smithi016.stdout: "source_pgid": "0.0", 2020-03-11T22:40:52.276 INFO:tasks.workunit.client.0.smithi016.stdout: "ready_epoch": 0, 2020-03-11T22:40:52.276 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_started": 0, 2020-03-11T22:40:52.277 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_clean": 0, 2020-03-11T22:40:52.277 INFO:tasks.workunit.client.0.smithi016.stdout: "source_version": "0'0", 2020-03-11T22:40:52.277 INFO:tasks.workunit.client.0.smithi016.stdout: "target_version": "0'0" 2020-03-11T22:40:52.277 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.277 INFO:tasks.workunit.client.0.smithi016.stdout: "last_change": "39", 2020-03-11T22:40:52.277 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend": "0", 2020-03-11T22:40:52.277 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:40:52.278 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_preluminous": "39", 2020-03-11T22:40:52.278 INFO:tasks.workunit.client.0.smithi016.stdout: "auid": 0, 2020-03-11T22:40:52.278 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:40:52.278 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_seq": 0, 2020-03-11T22:40:52.278 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_epoch": 0, 2020-03-11T22:40:52.278 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_snaps": [], 2020-03-11T22:40:52.278 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps": "[]", 2020-03-11T22:40:52.278 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_bytes": 0, 2020-03-11T22:40:52.279 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_objects": 0, 2020-03-11T22:40:52.279 INFO:tasks.workunit.client.0.smithi016.stdout: "tiers": [], 2020-03-11T22:40:52.279 INFO:tasks.workunit.client.0.smithi016.stdout: "tier_of": -1, 2020-03-11T22:40:52.279 INFO:tasks.workunit.client.0.smithi016.stdout: "read_tier": -1, 2020-03-11T22:40:52.279 INFO:tasks.workunit.client.0.smithi016.stdout: "write_tier": -1, 2020-03-11T22:40:52.279 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_mode": "none", 2020-03-11T22:40:52.279 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_bytes": 0, 2020-03-11T22:40:52.279 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_objects": 0, 2020-03-11T22:40:52.280 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:40:52.280 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:40:52.280 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:40:52.280 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_flush_age": 0, 2020-03-11T22:40:52.280 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_evict_age": 0, 2020-03-11T22:40:52.280 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profile": "", 2020-03-11T22:40:52.280 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_params": { 2020-03-11T22:40:52.281 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "none" 2020-03-11T22:40:52.281 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.281 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_period": 0, 2020-03-11T22:40:52.281 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_count": 0, 2020-03-11T22:40:52.281 INFO:tasks.workunit.client.0.smithi016.stdout: "use_gmt_hitset": true, 2020-03-11T22:40:52.281 INFO:tasks.workunit.client.0.smithi016.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:40:52.281 INFO:tasks.workunit.client.0.smithi016.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:40:52.282 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:40:52.282 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:40:52.282 INFO:tasks.workunit.client.0.smithi016.stdout: "grade_table": [], 2020-03-11T22:40:52.282 INFO:tasks.workunit.client.0.smithi016.stdout: "stripe_width": 0, 2020-03-11T22:40:52.282 INFO:tasks.workunit.client.0.smithi016.stdout: "expected_num_objects": 0, 2020-03-11T22:40:52.282 INFO:tasks.workunit.client.0.smithi016.stdout: "fast_read": false, 2020-03-11T22:40:52.282 INFO:tasks.workunit.client.0.smithi016.stdout: "options": {}, 2020-03-11T22:40:52.282 INFO:tasks.workunit.client.0.smithi016.stdout: "application_metadata": {} 2020-03-11T22:40:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: "osds": [ 2020-03-11T22:40:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:40:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "1adeefef-8ac6-40d7-ba02-a2dc9602cac5", 2020-03-11T22:40:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:40:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:40:52.284 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:40:52.284 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:40:52.284 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 5, 2020-03-11T22:40:52.284 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 30, 2020-03-11T22:40:52.284 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 36, 2020-03-11T22:40:52.284 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 43, 2020-03-11T22:40:52.284 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 31, 2020-03-11T22:40:52.285 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:40:52.285 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:40:52.285 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.285 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.285 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.285 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:40:52.286 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:40:52.286 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.286 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.286 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.286 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:40:52.286 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:40:52.286 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.287 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.287 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.287 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:40:52.287 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.287 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.287 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.287 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:40:52.287 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:40:52.288 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.288 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.288 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.288 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:40:52.288 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:40:52.288 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.288 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.289 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.289 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:52.289 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.289 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.289 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.289 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:40:52.289 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:40:52.290 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.290 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.290 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.290 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:40:52.290 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:40:52.290 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.290 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.291 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.291 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:52.291 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.291 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.291 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.291 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:40:52.291 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:40:52.292 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.292 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.292 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.292 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:40:52.292 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:40:52.292 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.292 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.292 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.293 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6803/39346", 2020-03-11T22:40:52.293 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6805/39346", 2020-03-11T22:40:52.293 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39346", 2020-03-11T22:40:52.293 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39346", 2020-03-11T22:40:52.293 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:40:52.293 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:40:52.293 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:40:52.294 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.294 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.294 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.294 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:40:52.294 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "55b263f4-40f2-4800-a15b-74333e0f0c12", 2020-03-11T22:40:52.294 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 0, 2020-03-11T22:40:52.294 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:40:52.295 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:40:52.295 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:40:52.295 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 0, 2020-03-11T22:40:52.295 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 0, 2020-03-11T22:40:52.295 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 10, 2020-03-11T22:40:52.295 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 22, 2020-03-11T22:40:52.295 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 32, 2020-03-11T22:40:52.295 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:40:52.296 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:40:52.296 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.296 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.296 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.296 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:52.296 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:40:52.296 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.296 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.297 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.297 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:52.297 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:40:52.297 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.297 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.297 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.297 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:40:52.298 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.298 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.298 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.298 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:52.298 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:40:52.298 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.298 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.298 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.299 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:52.299 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:40:52.299 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.299 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.299 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.299 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:52.299 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.300 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.300 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.300 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:52.300 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:40:52.300 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.300 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.300 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.300 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:52.301 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:40:52.301 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.301 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.301 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.301 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:52.301 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.301 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.301 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.302 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:52.302 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:40:52.302 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.302 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.302 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.302 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:52.302 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:40:52.303 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.303 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.303 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.303 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/34899", 2020-03-11T22:40:52.303 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/34899", 2020-03-11T22:40:52.303 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34899", 2020-03-11T22:40:52.303 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34899", 2020-03-11T22:40:52.304 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:40:52.304 INFO:tasks.workunit.client.0.smithi016.stdout: "exists" 2020-03-11T22:40:52.304 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.304 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.304 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.304 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:40:52.304 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "50c76265-8eaa-44a7-8947-296ee0f56e9d", 2020-03-11T22:40:52.304 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:40:52.305 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:40:52.305 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:40:52.305 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:40:52.305 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 14, 2020-03-11T22:40:52.305 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 30, 2020-03-11T22:40:52.305 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 38, 2020-03-11T22:40:52.305 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 0, 2020-03-11T22:40:52.306 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 33, 2020-03-11T22:40:52.306 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:40:52.306 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:40:52.306 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.306 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.306 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.306 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:52.306 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:40:52.307 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.307 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.307 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.307 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:52.307 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:40:52.307 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.307 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.308 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.308 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:40:52.308 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.308 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.308 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.308 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:52.308 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:40:52.308 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.309 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.309 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.309 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:52.309 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:40:52.309 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.309 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.309 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.310 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:52.310 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.310 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.310 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.310 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:52.310 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:40:52.310 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.310 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.311 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.311 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:52.311 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:40:52.311 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.311 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.311 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.311 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:52.311 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:40:52.312 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.312 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:40:52.312 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:52.312 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:40:52.312 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.312 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.312 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:40:52.313 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:52.313 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:40:52.313 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.313 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.313 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.313 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/39540", 2020-03-11T22:40:52.313 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/39540", 2020-03-11T22:40:52.314 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39540", 2020-03-11T22:40:52.314 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39540", 2020-03-11T22:40:52.314 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:40:52.314 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:40:52.314 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:40:52.314 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.315 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.315 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.315 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_xinfo": [ 2020-03-11T22:40:52.315 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.315 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:40:52.315 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:40:07.998203+0000", 2020-03-11T22:40:52.315 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:40:52.315 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:40:52.316 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:40:52.316 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:40:52.316 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:15.794856+0000", 2020-03-11T22:40:52.316 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:40:52.316 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.316 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.316 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:40:52.317 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:40:08.564347+0000", 2020-03-11T22:40:52.317 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:40:52.317 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:40:52.317 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:40:52.317 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:40:52.317 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:22.306040+0000", 2020-03-11T22:40:52.317 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:40:52.317 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.318 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.318 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:40:52.318 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:40:09.115246+0000", 2020-03-11T22:40:52.318 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:40:52.318 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:40:52.318 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:40:52.318 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:40:52.319 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:28.873764+0000", 2020-03-11T22:40:52.319 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:40:52.319 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.319 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.319 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap": [], 2020-03-11T22:40:52.319 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap_items": [], 2020-03-11T22:40:52.319 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_temp": [], 2020-03-11T22:40:52.319 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_temp": [], 2020-03-11T22:40:52.320 INFO:tasks.workunit.client.0.smithi016.stdout: "blacklist": {}, 2020-03-11T22:40:52.320 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profiles": { 2020-03-11T22:40:52.320 INFO:tasks.workunit.client.0.smithi016.stdout: "default": { 2020-03-11T22:40:52.320 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:40:52.320 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "2", 2020-03-11T22:40:52.320 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:40:52.320 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van" 2020-03-11T22:40:52.321 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.321 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.321 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps_queue": [], 2020-03-11T22:40:52.321 INFO:tasks.workunit.client.0.smithi016.stdout: "new_removed_snaps": [], 2020-03-11T22:40:52.321 INFO:tasks.workunit.client.0.smithi016.stdout: "new_purged_snaps": [], 2020-03-11T22:40:52.321 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_node_flags": {}, 2020-03-11T22:40:52.321 INFO:tasks.workunit.client.0.smithi016.stdout: "device_class_flags": {} 2020-03-11T22:40:52.321 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.322 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_metadata": [ 2020-03-11T22:40:52.322 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.322 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:40:52.322 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:40:52.322 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6804/39346,v1:127.0.0.1:6805/39346]", 2020-03-11T22:40:52.322 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:40:52.322 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:40:52.323 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:52.323 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:52.323 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:52.323 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:52.323 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:52.323 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:52.323 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:52.323 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:40:52.324 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:52.324 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:52.324 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:52.324 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:52.324 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:40:52.324 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:52.324 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:52.324 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:52.325 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:40:52.325 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:40:52.325 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:52.325 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:40:52.325 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:40:52.325 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:52.325 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:40:52.326 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6802/39346,v1:127.0.0.1:6803/39346]", 2020-03-11T22:40:52.326 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:40:52.326 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39346,v1:127.0.0.1:6809/39346]", 2020-03-11T22:40:52.326 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39346,v1:127.0.0.1:6807/39346]", 2020-03-11T22:40:52.326 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:40:52.326 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:40:52.326 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:52.327 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:52.327 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:52.327 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:40:52.327 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:52.327 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:52.327 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:40:52.327 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:40:52.327 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:52.328 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:40:52.328 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.328 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.328 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:40:52.328 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:40:52.328 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/34899,v1:127.0.0.1:6813/34899]", 2020-03-11T22:40:52.328 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:40:52.329 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:40:52.329 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:52.329 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:52.329 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:52.329 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:52.329 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:52.329 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:52.329 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:52.330 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:40:52.330 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:52.330 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:52.330 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:52.330 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:52.330 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:40:52.330 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:52.331 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:52.331 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:52.331 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:40:52.331 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:40:52.331 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:52.331 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:40:52.331 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:40:52.331 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:52.332 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:40:52.332 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/34899,v1:127.0.0.1:6811/34899]", 2020-03-11T22:40:52.332 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:40:52.332 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34899,v1:127.0.0.1:6817/34899]", 2020-03-11T22:40:52.332 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34899,v1:127.0.0.1:6815/34899]", 2020-03-11T22:40:52.332 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:40:52.332 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:40:52.333 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:52.333 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:52.333 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:52.333 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:40:52.333 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:52.333 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:52.333 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:40:52.334 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:40:52.334 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:52.334 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:40:52.334 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.334 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.334 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:40:52.334 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:40:52.334 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/39540,v1:127.0.0.1:6813/39540]", 2020-03-11T22:40:52.335 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:40:52.335 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:40:52.335 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:52.335 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:52.335 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:52.335 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:52.335 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:52.336 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:52.336 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:52.336 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:40:52.336 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:52.336 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:52.336 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:52.336 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:52.336 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:40:52.337 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:52.337 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:52.337 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:52.337 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:40:52.337 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:40:52.337 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:52.337 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:40:52.338 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:40:52.338 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:52.338 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:40:52.338 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/39540,v1:127.0.0.1:6811/39540]", 2020-03-11T22:40:52.338 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:40:52.338 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39540,v1:127.0.0.1:6817/39540]", 2020-03-11T22:40:52.338 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39540,v1:127.0.0.1:6815/39540]", 2020-03-11T22:40:52.338 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:40:52.339 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:40:52.339 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:52.339 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:52.339 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:52.339 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:40:52.339 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:52.339 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:52.340 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:40:52.340 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:40:52.340 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:52.340 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:40:52.340 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.340 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.340 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_first_committed": 1, 2020-03-11T22:40:52.340 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_last_committed": 44, 2020-03-11T22:40:52.341 INFO:tasks.workunit.client.0.smithi016.stdout: "crushmap": { 2020-03-11T22:40:52.341 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": [ 2020-03-11T22:40:52.341 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.341 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:40:52.341 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.0", 2020-03-11T22:40:52.341 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:40:52.341 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.342 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.342 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:40:52.342 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.1", 2020-03-11T22:40:52.342 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:40:52.342 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.342 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.342 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:40:52.342 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.2", 2020-03-11T22:40:52.343 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:40:52.343 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.343 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.343 INFO:tasks.workunit.client.0.smithi016.stdout: "types": [ 2020-03-11T22:40:52.343 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.343 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 0, 2020-03-11T22:40:52.343 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd" 2020-03-11T22:40:52.343 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.344 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.344 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:40:52.344 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "host" 2020-03-11T22:40:52.344 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.344 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.344 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 2, 2020-03-11T22:40:52.344 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "chassis" 2020-03-11T22:40:52.345 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.345 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.345 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 3, 2020-03-11T22:40:52.345 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "rack" 2020-03-11T22:40:52.345 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.345 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.345 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 4, 2020-03-11T22:40:52.345 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "row" 2020-03-11T22:40:52.346 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.346 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.346 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 5, 2020-03-11T22:40:52.346 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pdu" 2020-03-11T22:40:52.346 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.346 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.346 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 6, 2020-03-11T22:40:52.347 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pod" 2020-03-11T22:40:52.347 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.347 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.347 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 7, 2020-03-11T22:40:52.347 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "room" 2020-03-11T22:40:52.347 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.347 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.347 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 8, 2020-03-11T22:40:52.348 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "datacenter" 2020-03-11T22:40:52.348 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.348 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.348 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 9, 2020-03-11T22:40:52.348 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "zone" 2020-03-11T22:40:52.348 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.348 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.348 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 10, 2020-03-11T22:40:52.349 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "region" 2020-03-11T22:40:52.349 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.349 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.349 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:40:52.349 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "root" 2020-03-11T22:40:52.349 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.350 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.350 INFO:tasks.workunit.client.0.smithi016.stdout: "buckets": [ 2020-03-11T22:40:52.350 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.350 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -1, 2020-03-11T22:40:52.350 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default", 2020-03-11T22:40:52.351 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:40:52.351 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:40:52.351 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:40:52.351 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:40:52.351 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:40:52.351 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:40:52.351 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.351 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:40:52.352 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:40:52.352 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:40:52.352 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.352 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.352 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.352 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.352 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -2, 2020-03-11T22:40:52.353 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default~hdd", 2020-03-11T22:40:52.353 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:40:52.353 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:40:52.353 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:40:52.353 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:40:52.353 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:40:52.353 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:40:52.353 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.354 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:40:52.354 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:40:52.354 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:40:52.354 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.354 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.354 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.354 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.355 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:40:52.355 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016", 2020-03-11T22:40:52.355 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:40:52.355 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:40:52.355 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:40:52.355 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:40:52.355 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:40:52.355 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:40:52.356 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.356 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:40:52.356 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:40:52.356 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:40:52.356 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.356 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.356 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:40:52.357 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:40:52.357 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:40:52.357 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.357 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.357 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:40:52.357 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:40:52.357 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:40:52.357 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.358 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.358 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.358 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.358 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:40:52.358 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016~hdd", 2020-03-11T22:40:52.358 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:40:52.358 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:40:52.358 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:40:52.359 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:40:52.359 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:40:52.359 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:40:52.359 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.359 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:40:52.359 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:40:52.359 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:40:52.360 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.360 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.360 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:40:52.360 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:40:52.360 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:40:52.360 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.360 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.361 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:40:52.361 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:40:52.361 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:40:52.361 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.361 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.362 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.362 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.362 INFO:tasks.workunit.client.0.smithi016.stdout: "rules": [ 2020-03-11T22:40:52.362 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.362 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 0, 2020-03-11T22:40:52.362 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "replicated_rule", 2020-03-11T22:40:52.362 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 0, 2020-03-11T22:40:52.363 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:40:52.363 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 1, 2020-03-11T22:40:52.363 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 10, 2020-03-11T22:40:52.363 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:40:52.363 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.363 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:40:52.363 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:40:52.364 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:40:52.364 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.364 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.364 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_firstn", 2020-03-11T22:40:52.364 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:40:52.364 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:40:52.364 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.364 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.365 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:40:52.365 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.365 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:40:52.365 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.365 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.365 INFO:tasks.workunit.client.0.smithi016.stdout: "tunables": { 2020-03-11T22:40:52.365 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_tries": 0, 2020-03-11T22:40:52.366 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:40:52.366 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_total_tries": 50, 2020-03-11T22:40:52.366 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:40:52.366 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:40:52.366 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_stable": 1, 2020-03-11T22:40:52.366 INFO:tasks.workunit.client.0.smithi016.stdout: "straw_calc_version": 1, 2020-03-11T22:40:52.366 INFO:tasks.workunit.client.0.smithi016.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:40:52.366 INFO:tasks.workunit.client.0.smithi016.stdout: "profile": "jewel", 2020-03-11T22:40:52.367 INFO:tasks.workunit.client.0.smithi016.stdout: "optimal_tunables": 1, 2020-03-11T22:40:52.367 INFO:tasks.workunit.client.0.smithi016.stdout: "legacy_tunables": 0, 2020-03-11T22:40:52.367 INFO:tasks.workunit.client.0.smithi016.stdout: "minimum_required_version": "jewel", 2020-03-11T22:40:52.367 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables": 1, 2020-03-11T22:40:52.367 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables2": 1, 2020-03-11T22:40:52.367 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v2_rules": 0, 2020-03-11T22:40:52.367 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables3": 1, 2020-03-11T22:40:52.368 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v3_rules": 0, 2020-03-11T22:40:52.368 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v4_buckets": 1, 2020-03-11T22:40:52.368 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables5": 1, 2020-03-11T22:40:52.368 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v5_rules": 0 2020-03-11T22:40:52.368 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.368 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_args": {} 2020-03-11T22:40:52.369 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.369 INFO:tasks.workunit.client.0.smithi016.stdout: "fsmap": { 2020-03-11T22:40:52.369 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:40:52.369 INFO:tasks.workunit.client.0.smithi016.stdout: "default_fscid": -1, 2020-03-11T22:40:52.369 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": { 2020-03-11T22:40:52.369 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": {}, 2020-03-11T22:40:52.369 INFO:tasks.workunit.client.0.smithi016.stdout: "ro_compat": {}, 2020-03-11T22:40:52.370 INFO:tasks.workunit.client.0.smithi016.stdout: "incompat": { 2020-03-11T22:40:52.370 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_1": "base v0.20", 2020-03-11T22:40:52.370 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:40:52.370 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:40:52.370 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:40:52.370 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:40:52.370 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:40:52.370 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_8": "no anchor table", 2020-03-11T22:40:52.371 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_9": "file layout v2", 2020-03-11T22:40:52.371 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:40:52.371 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.371 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.371 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_flags": { 2020-03-11T22:40:52.371 INFO:tasks.workunit.client.0.smithi016.stdout: "enable_multiple": false, 2020-03-11T22:40:52.371 INFO:tasks.workunit.client.0.smithi016.stdout: "ever_enabled_multiple": false 2020-03-11T22:40:52.372 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.372 INFO:tasks.workunit.client.0.smithi016.stdout: "standbys": [], 2020-03-11T22:40:52.372 INFO:tasks.workunit.client.0.smithi016.stdout: "filesystems": [] 2020-03-11T22:40:52.372 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.372 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:40:52.372 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:40:52.372 INFO:tasks.workunit.client.0.smithi016.stdout: "auth": { 2020-03-11T22:40:52.372 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:40:52.373 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 5, 2020-03-11T22:40:52.373 INFO:tasks.workunit.client.0.smithi016.stdout: "num_secrets": 3 2020-03-11T22:40:52.373 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.373 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 32, 2020-03-11T22:40:52.373 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_active": 32, 2020-03-11T22:40:52.373 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_unknown": 0, 2020-03-11T22:40:52.373 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osd": 3, 2020-03-11T22:40:52.373 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_sum": { 2020-03-11T22:40:52.374 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:40:52.374 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48300, 2020-03-11T22:40:52.374 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:40:52.374 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:40:52.374 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:40:52.374 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:52.374 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:40:52.375 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:52.375 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:52.375 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:52.375 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:52.375 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:40:52.375 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:40:52.375 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:40:52.375 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 100, 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 100, 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:52.377 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:40:52.377 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:52.377 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:52.377 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:40:52.377 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:40:52.377 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:40:52.377 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:40:52.377 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:40:52.378 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:52.378 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:52.378 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:52.378 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:52.378 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:52.378 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:52.378 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:52.379 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:52.379 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:52.379 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:40:52.379 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:40:52.379 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.379 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:40:52.379 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:40:52.379 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:40:52.380 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:40:52.380 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 0, 2020-03-11T22:40:52.380 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 0, 2020-03-11T22:40:52.380 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:40:52.380 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:52.380 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:40:52.380 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:40:52.381 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:40:52.381 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.381 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 512, 2020-03-11T22:40:52.381 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 512, 2020-03-11T22:40:52.381 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 64, 2020-03-11T22:40:52.381 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 64, 2020-03-11T22:40:52.381 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 0 2020-03-11T22:40:52.382 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.382 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum": { 2020-03-11T22:40:52.382 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:40:52.382 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:40:52.382 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 65, 2020-03-11T22:40:52.382 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:40:52.382 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:52.382 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:52.383 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:40:52.383 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3180608, 2020-03-11T22:40:52.383 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 34560, 2020-03-11T22:40:52.383 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 7, 2020-03-11T22:40:52.383 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145720, 2020-03-11T22:40:52.383 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311392192, 2020-03-11T22:40:52.383 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:40:52.384 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:40:52.384 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318865604608, 2020-03-11T22:40:52.384 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:52.384 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 35389440, 2020-03-11T22:40:52.384 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 445494, 2020-03-11T22:40:52.384 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:40:52.384 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:52.384 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:40:52.385 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 7824, 2020-03-11T22:40:52.385 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221217648 2020-03-11T22:40:52.385 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.385 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:40:52.385 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:52.385 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:52.385 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:52.386 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:40:52.386 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:40:52.386 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:40:52.386 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.386 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:40:52.386 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:52.387 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:52.387 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:52.387 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:40:52.387 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.387 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:40:52.387 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:40:52.387 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.387 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum_by_class": { 2020-03-11T22:40:52.388 INFO:tasks.workunit.client.0.smithi016.stdout: "hdd": { 2020-03-11T22:40:52.388 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:40:52.388 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:40:52.388 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 65, 2020-03-11T22:40:52.388 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:40:52.388 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:52.388 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:52.389 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:40:52.389 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3180608, 2020-03-11T22:40:52.389 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 34560, 2020-03-11T22:40:52.389 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 7, 2020-03-11T22:40:52.389 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145720, 2020-03-11T22:40:52.389 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311392192, 2020-03-11T22:40:52.389 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:40:52.389 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:40:52.390 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318865604608, 2020-03-11T22:40:52.390 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:52.390 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 35389440, 2020-03-11T22:40:52.390 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 445494, 2020-03-11T22:40:52.390 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:40:52.390 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:52.390 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:40:52.391 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 7824, 2020-03-11T22:40:52.391 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221217648 2020-03-11T22:40:52.391 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.391 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:40:52.391 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:52.391 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:52.392 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:52.392 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:40:52.392 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:40:52.392 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:40:52.392 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.392 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:40:52.392 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:52.392 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:52.393 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:52.393 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:40:52.393 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.393 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:40:52.393 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:40:52.393 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.393 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.394 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_stats": [ 2020-03-11T22:40:52.394 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.394 INFO:tasks.workunit.client.0.smithi016.stdout: "poolid": 1, 2020-03-11T22:40:52.394 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 32, 2020-03-11T22:40:52.394 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:40:52.394 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48300, 2020-03-11T22:40:52.394 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:40:52.394 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:40:52.395 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:40:52.395 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:52.395 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:40:52.395 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:52.395 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:52.395 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:52.395 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:52.396 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:40:52.396 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:40:52.396 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:40:52.396 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 100, 2020-03-11T22:40:52.396 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 100, 2020-03-11T22:40:52.396 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:52.396 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:52.396 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:52.397 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:52.397 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:52.397 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:52.397 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:40:52.397 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:52.397 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:52.397 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:40:52.398 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:40:52.398 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:40:52.398 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:40:52.398 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:40:52.398 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:52.398 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:52.398 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:52.398 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:52.399 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:52.399 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:52.399 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:52.399 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:52.399 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:52.399 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:40:52.399 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:40:52.400 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.400 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:40:52.400 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:40:52.400 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:40:52.400 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:40:52.400 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 19529728, 2020-03-11T22:40:52.400 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 143228, 2020-03-11T22:40:52.400 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:40:52.401 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:52.401 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:40:52.401 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:40:52.401 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:40:52.401 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.401 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 512, 2020-03-11T22:40:52.401 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 512, 2020-03-11T22:40:52.401 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 64, 2020-03-11T22:40:52.402 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 64, 2020-03-11T22:40:52.402 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 3 2020-03-11T22:40:52.402 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.402 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.402 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_stats": [ 2020-03-11T22:40:52.402 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.402 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:40:52.403 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 154618822666 2020-03-11T22:40:52.403 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.403 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.403 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:40:52.403 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 42949672976 2020-03-11T22:40:52.403 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.403 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.403 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:40:52.404 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 163208757258 2020-03-11T22:40:52.404 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.404 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.404 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_state": [ 2020-03-11T22:40:52.404 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.404 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "active+undersized+degraded", 2020-03-11T22:40:52.404 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 32 2020-03-11T22:40:52.405 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.405 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.405 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_osd": [ 2020-03-11T22:40:52.405 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.405 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:40:52.405 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:40:52.405 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 32, 2020-03-11T22:40:52.405 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:52.406 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.406 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.406 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:40:52.406 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:40:52.406 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 0, 2020-03-11T22:40:52.406 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:52.406 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.406 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.407 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:40:52.407 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 32, 2020-03-11T22:40:52.407 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 32, 2020-03-11T22:40:52.407 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:52.407 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.407 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.407 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": [ 2020-03-11T22:40:52.408 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:40:52.408 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:40:52.408 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": {} 2020-03-11T22:40:52.408 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.408 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:40:52.408 INFO:tasks.workunit.client.0.smithi016.stdout: "servicemap": { 2020-03-11T22:40:52.408 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:40:52.408 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:39:08.641710+0000", 2020-03-11T22:40:52.409 INFO:tasks.workunit.client.0.smithi016.stdout: "services": {} 2020-03-11T22:40:52.409 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:40:52.409 INFO:tasks.workunit.client.0.smithi016.stdout: "paxos": { 2020-03-11T22:40:52.409 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:40:52.409 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 322, 2020-03-11T22:40:52.409 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pn": 0, 2020-03-11T22:40:52.409 INFO:tasks.workunit.client.0.smithi016.stdout: "accepted_pn": 0 2020-03-11T22:40:52.409 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:40:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:40:52.411 INFO:tasks.workunit.client.0.smithi016.stderr:report 3835489981 2020-03-11T22:40:52.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:40:52.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-11T22:40:52.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-11T22:40:52.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-11T22:40:52.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:writing non-divergent object existing_4 2020-03-11T22:40:52.599 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:52.600 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.290383+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.600 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274644+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.600 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289456+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.600 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289693+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.600 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289878+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.600 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272548+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.600 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.222288+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.601 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.288961+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.601 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289007+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.601 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289233+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.601 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273910+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.601 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272802+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.601 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274300+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.601 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274552+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.602 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274831+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.602 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274550+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.602 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273364+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.602 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273669+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.602 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273246+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.602 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274123+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.602 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289135+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.603 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289037+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.603 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289518+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.603 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272568+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.603 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272564+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.603 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272542+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.603 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289654+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.603 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273250+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.604 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273249+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.604 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273854+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.604 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273038+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.604 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273942+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:40:52.604 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:40:52.604 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:52.605 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:40:52.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:40:52.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:52.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-11T22:40:52.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:52.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:52.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:52.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:52.655 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:52.656 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:52.656 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:52.657 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:52.657 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:52.657 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:52.715 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:52.715 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:52.715 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:52.715 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:52.715 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:53.089 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:53.090 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:40:53.090 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:40:53.090 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:53.090 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:53.090 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:53.225 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822669 2020-03-11T22:40:53.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822669' 2020-03-11T22:40:53.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:53.226 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:53.322 INFO:tasks.workunit.client.0.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:40:53.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:40:53.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822669 1-' 2020-03-11T22:40:53.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:53.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:53.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=163208757260 2020-03-11T22:40:53.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822669 1- 2-163208757260' 2020-03-11T22:40:53.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:53.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822669 2020-03-11T22:40:53.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:53.468 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:53.469 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822669 2020-03-11T22:40:53.469 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:53.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822669 2020-03-11T22:40:53.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822669' 2020-03-11T22:40:53.470 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:53.470 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 154618822669 2020-03-11T22:40:53.832 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822667 -lt 154618822669 2020-03-11T22:40:53.832 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:54.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:54.833 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:55.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822669 -lt 154618822669 2020-03-11T22:40:55.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:55.210 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:40:55.211 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:55.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:55.212 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:40:55.212 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:55.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:40:55.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:40:55.213 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:55.214 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 2020-03-11T22:40:55.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 2020-03-11T22:40:55.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672976: unary operator expected 2020-03-11T22:40:55.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:55.594 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-163208757260 2020-03-11T22:40:55.594 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:55.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:55.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-163208757260 2020-03-11T22:40:55.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:55.596 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=163208757260 2020-03-11T22:40:55.597 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 163208757260' 2020-03-11T22:40:55.597 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:55.597 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 163208757260 2020-03-11T22:40:55.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 163208757261 -lt 163208757260 2020-03-11T22:40:55.984 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:55.985 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:55.985 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:56.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:40:56.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:56.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:56.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:56.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:56.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:56.476 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:56.476 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:56.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:56.835 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:56.835 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:56.835 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:57.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:57.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:40:57.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:40:57.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:40:57.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:57.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:40:57.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:57.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:57.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:57.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:57.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:57.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:57.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:57.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:57.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:57.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:57.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:57.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:58.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:58.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:58.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:58.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:58.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:58.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:58.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:58.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:58.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:58.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:58.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:58.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:40:58.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:58.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:40:58.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:58.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:58.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:58.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:58.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:58.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:58.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:58.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:59.293 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:59.293 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:59.294 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:59.294 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:59.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:59.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:59.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:59.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:59.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:59.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:59.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:59.763 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:59.763 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:00.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:00.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:00.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:41:00.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:00.236 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:41:00.636 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:00.636 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:00.637 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:00.637 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:00.637 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:00.637 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:00.637 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:00.637 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:00.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:00.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:00.997 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:00.997 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:01.469 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:01.469 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:01.469 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:01.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:01.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:01.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:01.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:01.471 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:01.471 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:01.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:01.938 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:01.938 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:41:01.938 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:01.938 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:41:02.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:02.739 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:02.739 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:02.740 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:02.740 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:02.740 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:02.740 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:02.740 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:03.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:03.108 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:03.108 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:03.108 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:03.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:03.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:03.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:03.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:03.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:03.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:03.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:03.579 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:03.580 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:04.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:04.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:04.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:41:04.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:04.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:41:05.646 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:05.646 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:05.648 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:05.648 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:05.648 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:05.648 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:05.649 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:05.649 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:06.014 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:06.015 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:06.015 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:06.015 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:06.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:06.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:06.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:06.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:06.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:06.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:06.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:06.490 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:06.490 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:06.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:06.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:06.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:41:06.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:06.968 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:41:10.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:10.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:10.169 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:10.169 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:10.170 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:10.170 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:10.170 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:10.170 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:10.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:10.534 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:10.534 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:10.535 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:11.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:11.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:11.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:11.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:11.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:11.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:11.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:11.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:11.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:11.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:11.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:11.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:41:11.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:11.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:41:17.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:17.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:17.887 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:17.887 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:17.888 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:17.888 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:17.888 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:17.888 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:18.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:18.255 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:18.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:18.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:18.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:18.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:18.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:18.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:18.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:18.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:18.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:18.728 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:18.728 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:19.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:19.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:19.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:41:19.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:19.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:41:26.504 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:26.504 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:26.505 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:26.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:26.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:26.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:26.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:26.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:26.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:26.876 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:26.876 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:26.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:27.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:27.351 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:27.351 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:27.351 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:27.351 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:27.351 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:27.351 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:27.352 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:27.352 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:27.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:27.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:27.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:41:27.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:41:28.430 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:41:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_fingerprint": "3f106c27-51f9-4316-a5f5-e19ac3bff60f", 2020-03-11T22:41:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:41:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:41:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "timestamp": "2020-03-11T22:41:28.426896+0000", 2020-03-11T22:41:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "tag": "", 2020-03-11T22:41:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "health": { 2020-03-11T22:41:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "status": "HEALTH_WARN", 2020-03-11T22:41:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "checks": { 2020-03-11T22:41:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:41:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:41:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:41:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 17 2020-03-11T22:41:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [], 2020-03-11T22:41:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:41:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "OSD_DOWN": { 2020-03-11T22:41:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:41:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 osds down", 2020-03-11T22:41:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:41:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:41:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "osd.1 (root=default,host=smithi016) is down" 2020-03-11T22:41:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:41:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: "PG_DEGRADED": { 2020-03-11T22:41:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:41:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-03-11T22:41:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 64 2020-03-11T22:41:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:41:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.0 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.2 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.3 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.4 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.5 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.6 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.7 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.8 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.9 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.a is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.b is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.c is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.d is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.e is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.f is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.10 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.11 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.12 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.13 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.14 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.15 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.16 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.17 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.18 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.19 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1a is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1b is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1c is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1d is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1e is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.1f is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:41:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:41:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:41:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:41:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:41:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:41:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:41:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:41:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:41:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: "mutes": [] 2020-03-11T22:41:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_first_committed": 1, 2020-03-11T22:41:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_last_committed": 1, 2020-03-11T22:41:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap": { 2020-03-11T22:41:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:41:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:41:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:39:07.882909Z", 2020-03-11T22:41:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:39:07.882909Z", 2020-03-11T22:41:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release": 15, 2020-03-11T22:41:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:41:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "features": { 2020-03-11T22:41:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "persistent": [ 2020-03-11T22:41:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "kraken", 2020-03-11T22:41:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "luminous", 2020-03-11T22:41:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "mimic", 2020-03-11T22:41:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap-prune", 2020-03-11T22:41:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "nautilus", 2020-03-11T22:41:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "octopus" 2020-03-11T22:41:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "optional": [] 2020-03-11T22:41:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "mons": [ 2020-03-11T22:41:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "rank": 0, 2020-03-11T22:41:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "a", 2020-03-11T22:41:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:41:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:41:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 0 2020-03-11T22:41:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:41:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:41:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 0, 2020-03-11T22:41:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 0 2020-03-11T22:41:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "quorum": [ 2020-03-11T22:41:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: 0 2020-03-11T22:41:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap": { 2020-03-11T22:41:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 44, 2020-03-11T22:41:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:41:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:39:08.641393+0000", 2020-03-11T22:41:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:40:16.442437+0000", 2020-03-11T22:41:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "last_up_change": "2020-03-11T22:40:14.553542+0000", 2020-03-11T22:41:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "last_in_change": "2020-03-11T22:40:16.221254+0000", 2020-03-11T22:41:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:41:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_num": 5800384, 2020-03-11T22:41:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_set": [ 2020-03-11T22:41:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "nodown", 2020-03-11T22:41:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "noin", 2020-03-11T22:41:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "noout", 2020-03-11T22:41:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "pglog_hardlimit", 2020-03-11T22:41:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snapdirs", 2020-03-11T22:41:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "recovery_deletes", 2020-03-11T22:41:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "sortbitwise" 2020-03-11T22:41:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_version": 7, 2020-03-11T22:41:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:41:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:41:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:41:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_snapshot": "", 2020-03-11T22:41:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_max": 1, 2020-03-11T22:41:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: "max_osd": 3, 2020-03-11T22:41:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:41:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: "min_compat_client": "jewel", 2020-03-11T22:41:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: "require_osd_release": "octopus", 2020-03-11T22:41:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: "pools": [ 2020-03-11T22:41:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:41:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_name": "test", 2020-03-11T22:41:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "create_time": "2020-03-11T22:39:31.702645+0000", 2020-03-11T22:41:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": 1, 2020-03-11T22:41:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_names": "hashpspool", 2020-03-11T22:41:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:41:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "size": 3, 2020-03-11T22:41:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 2, 2020-03-11T22:41:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_rule": 0, 2020-03-11T22:41:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "object_hash": 2, 2020-03-11T22:41:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:41:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num": 32, 2020-03-11T22:41:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num": 1, 2020-03-11T22:41:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num_target": 32, 2020-03-11T22:41:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_target": 32, 2020-03-11T22:41:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_pending": 32, 2020-03-11T22:41:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pg_merge_meta": { 2020-03-11T22:41:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "source_pgid": "0.0", 2020-03-11T22:41:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "ready_epoch": 0, 2020-03-11T22:41:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_started": 0, 2020-03-11T22:41:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_clean": 0, 2020-03-11T22:41:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "source_version": "0'0", 2020-03-11T22:41:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "target_version": "0'0" 2020-03-11T22:41:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "last_change": "39", 2020-03-11T22:41:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend": "0", 2020-03-11T22:41:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:41:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_preluminous": "39", 2020-03-11T22:41:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "auid": 0, 2020-03-11T22:41:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:41:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_seq": 0, 2020-03-11T22:41:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_epoch": 0, 2020-03-11T22:41:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_snaps": [], 2020-03-11T22:41:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps": "[]", 2020-03-11T22:41:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_bytes": 0, 2020-03-11T22:41:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_objects": 0, 2020-03-11T22:41:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "tiers": [], 2020-03-11T22:41:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "tier_of": -1, 2020-03-11T22:41:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "read_tier": -1, 2020-03-11T22:41:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "write_tier": -1, 2020-03-11T22:41:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_mode": "none", 2020-03-11T22:41:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_bytes": 0, 2020-03-11T22:41:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_objects": 0, 2020-03-11T22:41:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:41:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:41:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:41:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_flush_age": 0, 2020-03-11T22:41:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_evict_age": 0, 2020-03-11T22:41:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profile": "", 2020-03-11T22:41:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_params": { 2020-03-11T22:41:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "none" 2020-03-11T22:41:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_period": 0, 2020-03-11T22:41:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_count": 0, 2020-03-11T22:41:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "use_gmt_hitset": true, 2020-03-11T22:41:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:41:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:41:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:41:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:41:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "grade_table": [], 2020-03-11T22:41:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "stripe_width": 0, 2020-03-11T22:41:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "expected_num_objects": 0, 2020-03-11T22:41:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: "fast_read": false, 2020-03-11T22:41:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: "options": {}, 2020-03-11T22:41:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: "application_metadata": {} 2020-03-11T22:41:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: "osds": [ 2020-03-11T22:41:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:41:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "1adeefef-8ac6-40d7-ba02-a2dc9602cac5", 2020-03-11T22:41:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:41:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:41:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:41:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:41:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 5, 2020-03-11T22:41:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 30, 2020-03-11T22:41:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 36, 2020-03-11T22:41:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 43, 2020-03-11T22:41:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 31, 2020-03-11T22:41:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:41:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:41:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:41:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:41:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:41:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:41:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:41:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:41:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:41:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:41:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:41:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:41:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:41:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:41:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:41:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:41:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:41:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:41:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39346 2020-03-11T22:41:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6803/39346", 2020-03-11T22:41:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6805/39346", 2020-03-11T22:41:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39346", 2020-03-11T22:41:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39346", 2020-03-11T22:41:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:41:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:41:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:41:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:41:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "55b263f4-40f2-4800-a15b-74333e0f0c12", 2020-03-11T22:41:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 0, 2020-03-11T22:41:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:41:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:41:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:41:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 0, 2020-03-11T22:41:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 0, 2020-03-11T22:41:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 10, 2020-03-11T22:41:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 22, 2020-03-11T22:41:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 32, 2020-03-11T22:41:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:41:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:41:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:41:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:41:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:41:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:41:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:41:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:41:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:41:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:41:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 34899 2020-03-11T22:41:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/34899", 2020-03-11T22:41:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/34899", 2020-03-11T22:41:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34899", 2020-03-11T22:41:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34899", 2020-03-11T22:41:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:41:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "exists" 2020-03-11T22:41:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:41:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "50c76265-8eaa-44a7-8947-296ee0f56e9d", 2020-03-11T22:41:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:41:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:41:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:41:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:41:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 14, 2020-03-11T22:41:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 30, 2020-03-11T22:41:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 38, 2020-03-11T22:41:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 0, 2020-03-11T22:41:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 33, 2020-03-11T22:41:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:41:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:41:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:41:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:41:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:41:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:41:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:41:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:41:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:41:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:41:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:41:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:41:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:41:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 39540 2020-03-11T22:41:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/39540", 2020-03-11T22:41:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/39540", 2020-03-11T22:41:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39540", 2020-03-11T22:41:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39540", 2020-03-11T22:41:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:41:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:41:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:41:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_xinfo": [ 2020-03-11T22:41:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:41:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:40:07.998203+0000", 2020-03-11T22:41:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:41:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:41:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:41:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:41:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:15.794856+0000", 2020-03-11T22:41:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:41:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:41:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:40:08.564347+0000", 2020-03-11T22:41:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:41:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:41:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:41:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:41:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:22.306040+0000", 2020-03-11T22:41:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:41:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:41:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:40:09.115246+0000", 2020-03-11T22:41:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:41:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:41:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:41:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:41:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:28.873764+0000", 2020-03-11T22:41:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:41:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap": [], 2020-03-11T22:41:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap_items": [], 2020-03-11T22:41:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_temp": [], 2020-03-11T22:41:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_temp": [], 2020-03-11T22:41:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "blacklist": {}, 2020-03-11T22:41:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profiles": { 2020-03-11T22:41:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "default": { 2020-03-11T22:41:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:41:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "2", 2020-03-11T22:41:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:41:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van" 2020-03-11T22:41:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps_queue": [], 2020-03-11T22:41:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: "new_removed_snaps": [], 2020-03-11T22:41:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: "new_purged_snaps": [], 2020-03-11T22:41:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_node_flags": {}, 2020-03-11T22:41:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: "device_class_flags": {} 2020-03-11T22:41:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_metadata": [ 2020-03-11T22:41:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:41:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:41:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6804/39346,v1:127.0.0.1:6805/39346]", 2020-03-11T22:41:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:41:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:41:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:41:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:41:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:41:28.520 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:41:28.520 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:41:28.520 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:28.520 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:41:28.520 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:41:28.520 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:28.520 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:41:28.521 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6802/39346,v1:127.0.0.1:6803/39346]", 2020-03-11T22:41:28.521 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:41:28.521 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39346,v1:127.0.0.1:6809/39346]", 2020-03-11T22:41:28.521 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39346,v1:127.0.0.1:6807/39346]", 2020-03-11T22:41:28.521 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:41:28.521 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:41:28.521 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:28.522 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:28.522 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:28.522 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:41:28.522 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:28.522 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:28.522 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:41:28.522 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:41:28.523 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:28.523 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:41:28.523 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.523 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.523 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:41:28.523 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:41:28.523 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/34899,v1:127.0.0.1:6813/34899]", 2020-03-11T22:41:28.523 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:41:28.524 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:41:28.524 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:28.524 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:28.524 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:28.524 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:28.524 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:28.524 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:28.525 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:28.525 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:41:28.525 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:28.525 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:28.525 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:28.525 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:28.525 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:41:28.526 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:28.526 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:28.526 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:41:28.526 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:41:28.526 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:41:28.526 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:28.526 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:41:28.527 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:41:28.527 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:28.527 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:41:28.527 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/34899,v1:127.0.0.1:6811/34899]", 2020-03-11T22:41:28.527 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:41:28.527 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34899,v1:127.0.0.1:6817/34899]", 2020-03-11T22:41:28.527 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34899,v1:127.0.0.1:6815/34899]", 2020-03-11T22:41:28.528 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:41:28.528 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:41:28.528 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:28.528 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:28.528 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:28.528 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:41:28.529 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:28.529 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:28.529 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:41:28.529 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:41:28.529 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:28.529 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:41:28.529 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.530 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.530 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:41:28.530 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:41:28.530 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/39540,v1:127.0.0.1:6813/39540]", 2020-03-11T22:41:28.530 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:41:28.530 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:41:28.530 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:28.531 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:28.531 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:28.531 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:28.531 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:28.531 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:28.531 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:28.531 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:41:28.532 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:28.532 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:28.532 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:28.532 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:28.532 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:41:28.532 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:28.532 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:28.533 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:41:28.533 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:41:28.533 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:41:28.533 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:28.533 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:41:28.533 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:41:28.533 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:28.533 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:41:28.534 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/39540,v1:127.0.0.1:6811/39540]", 2020-03-11T22:41:28.534 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:41:28.534 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39540,v1:127.0.0.1:6817/39540]", 2020-03-11T22:41:28.534 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39540,v1:127.0.0.1:6815/39540]", 2020-03-11T22:41:28.534 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:41:28.534 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:41:28.534 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:28.535 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:28.535 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:28.535 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:41:28.535 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:28.535 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:28.535 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:41:28.535 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:41:28.536 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:28.536 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:41:28.536 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.536 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.536 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_first_committed": 1, 2020-03-11T22:41:28.536 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_last_committed": 44, 2020-03-11T22:41:28.536 INFO:tasks.workunit.client.0.smithi016.stdout: "crushmap": { 2020-03-11T22:41:28.536 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": [ 2020-03-11T22:41:28.537 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.537 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:41:28.537 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.0", 2020-03-11T22:41:28.537 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:41:28.537 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.537 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.537 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:41:28.538 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.1", 2020-03-11T22:41:28.538 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:41:28.538 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.538 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.538 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:41:28.539 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.2", 2020-03-11T22:41:28.539 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:41:28.539 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.539 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.539 INFO:tasks.workunit.client.0.smithi016.stdout: "types": [ 2020-03-11T22:41:28.539 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.539 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 0, 2020-03-11T22:41:28.540 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd" 2020-03-11T22:41:28.540 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.540 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.540 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:41:28.540 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "host" 2020-03-11T22:41:28.540 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.541 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.541 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 2, 2020-03-11T22:41:28.541 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "chassis" 2020-03-11T22:41:28.541 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.541 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.541 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 3, 2020-03-11T22:41:28.541 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "rack" 2020-03-11T22:41:28.542 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.542 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.542 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 4, 2020-03-11T22:41:28.542 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "row" 2020-03-11T22:41:28.542 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.542 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.542 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 5, 2020-03-11T22:41:28.543 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pdu" 2020-03-11T22:41:28.543 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.543 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.543 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 6, 2020-03-11T22:41:28.543 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pod" 2020-03-11T22:41:28.543 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.543 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.543 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 7, 2020-03-11T22:41:28.544 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "room" 2020-03-11T22:41:28.544 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.544 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.544 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 8, 2020-03-11T22:41:28.544 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "datacenter" 2020-03-11T22:41:28.544 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.545 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.545 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 9, 2020-03-11T22:41:28.545 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "zone" 2020-03-11T22:41:28.545 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.545 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.545 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 10, 2020-03-11T22:41:28.545 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "region" 2020-03-11T22:41:28.546 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.546 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.546 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:41:28.546 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "root" 2020-03-11T22:41:28.546 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.546 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.546 INFO:tasks.workunit.client.0.smithi016.stdout: "buckets": [ 2020-03-11T22:41:28.547 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.547 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -1, 2020-03-11T22:41:28.547 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default", 2020-03-11T22:41:28.547 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:41:28.547 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:41:28.547 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:41:28.547 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:41:28.548 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:41:28.548 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:41:28.548 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.548 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:41:28.548 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:41:28.548 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:41:28.549 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.549 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.549 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.549 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.549 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -2, 2020-03-11T22:41:28.549 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default~hdd", 2020-03-11T22:41:28.549 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:41:28.550 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:41:28.550 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:41:28.550 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:41:28.550 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:41:28.550 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:41:28.550 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.550 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:41:28.551 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:41:28.551 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:41:28.551 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.551 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.551 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.551 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.551 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:41:28.552 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016", 2020-03-11T22:41:28.552 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:41:28.552 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:41:28.552 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:41:28.552 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:41:28.552 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:41:28.552 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:41:28.553 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.553 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:41:28.553 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:41:28.553 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:41:28.553 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.553 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.553 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:41:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:41:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:41:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:41:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:41:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:41:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:41:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016~hdd", 2020-03-11T22:41:28.556 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:41:28.556 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:41:28.556 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:41:28.556 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:41:28.556 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:41:28.556 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:41:28.556 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.557 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:41:28.557 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:41:28.557 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:41:28.557 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.557 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.557 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:41:28.557 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:41:28.558 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:41:28.558 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.558 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.558 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:41:28.558 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:41:28.558 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:41:28.559 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.559 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.559 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.559 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.559 INFO:tasks.workunit.client.0.smithi016.stdout: "rules": [ 2020-03-11T22:41:28.559 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.559 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 0, 2020-03-11T22:41:28.560 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "replicated_rule", 2020-03-11T22:41:28.560 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 0, 2020-03-11T22:41:28.560 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:41:28.560 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 1, 2020-03-11T22:41:28.560 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 10, 2020-03-11T22:41:28.560 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:41:28.560 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.560 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:41:28.561 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:41:28.561 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:41:28.561 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.561 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.561 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_firstn", 2020-03-11T22:41:28.561 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:41:28.562 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:41:28.562 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.562 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.562 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:41:28.562 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.562 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:41:28.563 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.563 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.563 INFO:tasks.workunit.client.0.smithi016.stdout: "tunables": { 2020-03-11T22:41:28.563 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_tries": 0, 2020-03-11T22:41:28.563 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:41:28.563 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_total_tries": 50, 2020-03-11T22:41:28.563 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:41:28.564 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:41:28.564 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_stable": 1, 2020-03-11T22:41:28.564 INFO:tasks.workunit.client.0.smithi016.stdout: "straw_calc_version": 1, 2020-03-11T22:41:28.564 INFO:tasks.workunit.client.0.smithi016.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:41:28.564 INFO:tasks.workunit.client.0.smithi016.stdout: "profile": "jewel", 2020-03-11T22:41:28.565 INFO:tasks.workunit.client.0.smithi016.stdout: "optimal_tunables": 1, 2020-03-11T22:41:28.565 INFO:tasks.workunit.client.0.smithi016.stdout: "legacy_tunables": 0, 2020-03-11T22:41:28.565 INFO:tasks.workunit.client.0.smithi016.stdout: "minimum_required_version": "jewel", 2020-03-11T22:41:28.565 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables": 1, 2020-03-11T22:41:28.565 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables2": 1, 2020-03-11T22:41:28.565 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v2_rules": 0, 2020-03-11T22:41:28.565 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables3": 1, 2020-03-11T22:41:28.565 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v3_rules": 0, 2020-03-11T22:41:28.566 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v4_buckets": 1, 2020-03-11T22:41:28.566 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables5": 1, 2020-03-11T22:41:28.566 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v5_rules": 0 2020-03-11T22:41:28.566 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.566 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_args": {} 2020-03-11T22:41:28.566 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.566 INFO:tasks.workunit.client.0.smithi016.stdout: "fsmap": { 2020-03-11T22:41:28.567 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:41:28.567 INFO:tasks.workunit.client.0.smithi016.stdout: "default_fscid": -1, 2020-03-11T22:41:28.567 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": { 2020-03-11T22:41:28.567 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": {}, 2020-03-11T22:41:28.567 INFO:tasks.workunit.client.0.smithi016.stdout: "ro_compat": {}, 2020-03-11T22:41:28.567 INFO:tasks.workunit.client.0.smithi016.stdout: "incompat": { 2020-03-11T22:41:28.567 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_1": "base v0.20", 2020-03-11T22:41:28.568 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:41:28.568 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:41:28.568 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:41:28.568 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:41:28.568 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:41:28.568 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_8": "no anchor table", 2020-03-11T22:41:28.568 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_9": "file layout v2", 2020-03-11T22:41:28.568 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:41:28.569 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.569 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.569 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_flags": { 2020-03-11T22:41:28.569 INFO:tasks.workunit.client.0.smithi016.stdout: "enable_multiple": false, 2020-03-11T22:41:28.569 INFO:tasks.workunit.client.0.smithi016.stdout: "ever_enabled_multiple": false 2020-03-11T22:41:28.570 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.570 INFO:tasks.workunit.client.0.smithi016.stdout: "standbys": [], 2020-03-11T22:41:28.570 INFO:tasks.workunit.client.0.smithi016.stdout: "filesystems": [] 2020-03-11T22:41:28.570 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.570 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:41:28.570 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:41:28.570 INFO:tasks.workunit.client.0.smithi016.stdout: "auth": { 2020-03-11T22:41:28.571 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:41:28.571 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 5, 2020-03-11T22:41:28.571 INFO:tasks.workunit.client.0.smithi016.stdout: "num_secrets": 3 2020-03-11T22:41:28.571 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.571 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 32, 2020-03-11T22:41:28.571 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_active": 32, 2020-03-11T22:41:28.571 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_unknown": 0, 2020-03-11T22:41:28.572 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osd": 3, 2020-03-11T22:41:28.572 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_sum": { 2020-03-11T22:41:28.572 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:41:28.572 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 47947, 2020-03-11T22:41:28.572 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:41:28.572 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:41:28.572 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:41:28.572 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:28.573 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:41:28.573 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:28.573 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:28.573 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:28.573 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:28.573 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:41:28.574 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:41:28.574 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:41:28.574 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 101, 2020-03-11T22:41:28.574 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 101, 2020-03-11T22:41:28.574 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:28.574 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:28.575 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:28.575 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:28.575 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:28.575 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:28.575 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:41:28.575 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:28.575 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:28.575 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:41:28.576 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:41:28.576 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:41:28.576 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:41:28.576 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:41:28.576 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:28.576 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:28.576 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:28.577 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:28.577 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:28.577 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:28.577 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:28.577 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:28.577 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:28.578 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:41:28.578 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:41:28.578 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.578 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:41:28.578 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:41:28.578 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:41:28.578 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:41:28.579 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 0, 2020-03-11T22:41:28.579 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 0, 2020-03-11T22:41:28.579 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:41:28.579 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:28.579 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:41:28.580 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:41:28.580 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:41:28.580 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.580 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 513, 2020-03-11T22:41:28.580 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 513, 2020-03-11T22:41:28.580 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 64, 2020-03-11T22:41:28.580 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 64, 2020-03-11T22:41:28.581 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 0 2020-03-11T22:41:28.581 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.581 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum": { 2020-03-11T22:41:28.581 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:41:28.581 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:41:28.581 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 65, 2020-03-11T22:41:28.581 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:41:28.582 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:28.582 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:28.582 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:41:28.582 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3180608, 2020-03-11T22:41:28.582 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 34560, 2020-03-11T22:41:28.582 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 7, 2020-03-11T22:41:28.582 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145720, 2020-03-11T22:41:28.582 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311392192, 2020-03-11T22:41:28.583 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:41:28.583 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:41:28.583 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318865604608, 2020-03-11T22:41:28.583 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:28.583 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 35389440, 2020-03-11T22:41:28.583 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 444788, 2020-03-11T22:41:28.583 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:41:28.584 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:28.584 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:41:28.584 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 7824, 2020-03-11T22:41:28.584 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221217648 2020-03-11T22:41:28.584 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.584 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:41:28.584 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:28.585 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:28.585 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:28.585 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:41:28.585 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:41:28.585 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:41:28.585 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.585 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:41:28.585 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:28.586 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:28.586 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:28.586 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:41:28.586 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.586 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:41:28.586 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:41:28.586 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.587 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum_by_class": { 2020-03-11T22:41:28.587 INFO:tasks.workunit.client.0.smithi016.stdout: "hdd": { 2020-03-11T22:41:28.587 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:41:28.587 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:41:28.587 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 65, 2020-03-11T22:41:28.587 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:41:28.587 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:28.587 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:28.588 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:41:28.588 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3180608, 2020-03-11T22:41:28.588 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 34560, 2020-03-11T22:41:28.588 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 7, 2020-03-11T22:41:28.588 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145720, 2020-03-11T22:41:28.588 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311392192, 2020-03-11T22:41:28.588 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:41:28.589 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:41:28.589 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318865604608, 2020-03-11T22:41:28.589 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:28.589 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 35389440, 2020-03-11T22:41:28.589 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 444788, 2020-03-11T22:41:28.590 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:41:28.590 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:28.590 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:41:28.590 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 7824, 2020-03-11T22:41:28.590 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221217648 2020-03-11T22:41:28.590 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.590 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:41:28.590 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:28.591 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:28.591 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:28.591 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:41:28.591 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:41:28.591 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:41:28.591 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.591 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:41:28.592 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:28.592 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:28.592 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:28.592 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:41:28.592 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.592 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:41:28.592 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:41:28.592 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.593 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.593 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_stats": [ 2020-03-11T22:41:28.593 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.593 INFO:tasks.workunit.client.0.smithi016.stdout: "poolid": 1, 2020-03-11T22:41:28.593 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 32, 2020-03-11T22:41:28.593 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:41:28.593 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 47947, 2020-03-11T22:41:28.594 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 100, 2020-03-11T22:41:28.594 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 0, 2020-03-11T22:41:28.594 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 300, 2020-03-11T22:41:28.594 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:28.594 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:41:28.594 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:28.594 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:28.595 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:28.595 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:28.595 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:41:28.595 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:41:28.595 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:41:28.595 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 101, 2020-03-11T22:41:28.595 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 101, 2020-03-11T22:41:28.596 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:28.596 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:28.596 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:28.596 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:28.596 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:28.596 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:28.596 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:41:28.597 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:28.597 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:28.597 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:41:28.597 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:41:28.597 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:41:28.597 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:41:28.598 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:41:28.598 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:28.598 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:28.598 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:28.598 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:28.598 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:28.598 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:28.598 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:28.599 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:28.599 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:28.599 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:41:28.599 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:41:28.599 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.599 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:41:28.600 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:41:28.600 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:41:28.600 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:41:28.600 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 19529728, 2020-03-11T22:41:28.600 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 142522, 2020-03-11T22:41:28.600 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:41:28.601 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:28.601 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:41:28.601 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:41:28.601 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:41:28.601 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.601 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 513, 2020-03-11T22:41:28.601 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 513, 2020-03-11T22:41:28.601 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 64, 2020-03-11T22:41:28.602 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 64, 2020-03-11T22:41:28.602 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 3 2020-03-11T22:41:28.602 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.602 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.603 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_stats": [ 2020-03-11T22:41:28.604 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.604 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:41:28.604 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 154618822675 2020-03-11T22:41:28.604 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.604 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.604 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:41:28.604 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 42949672976 2020-03-11T22:41:28.605 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.605 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.605 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:41:28.605 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 163208757267 2020-03-11T22:41:28.605 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.605 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.605 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_state": [ 2020-03-11T22:41:28.606 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.606 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "active+undersized+degraded", 2020-03-11T22:41:28.606 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 32 2020-03-11T22:41:28.606 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.606 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.606 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_osd": [ 2020-03-11T22:41:28.606 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.607 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:41:28.607 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:41:28.607 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 32, 2020-03-11T22:41:28.607 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:28.607 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.607 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.607 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:41:28.608 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:41:28.608 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 0, 2020-03-11T22:41:28.608 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:28.608 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.608 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.608 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:41:28.608 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 32, 2020-03-11T22:41:28.608 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 32, 2020-03-11T22:41:28.609 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:28.609 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.609 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.609 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": [ 2020-03-11T22:41:28.609 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:41:28.609 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:41:28.609 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": {} 2020-03-11T22:41:28.610 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.610 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:41:28.610 INFO:tasks.workunit.client.0.smithi016.stdout: "servicemap": { 2020-03-11T22:41:28.610 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:41:28.610 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:39:08.641710+0000", 2020-03-11T22:41:28.611 INFO:tasks.workunit.client.0.smithi016.stdout: "services": {} 2020-03-11T22:41:28.611 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:41:28.611 INFO:tasks.workunit.client.0.smithi016.stdout: "paxos": { 2020-03-11T22:41:28.611 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:41:28.611 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 402, 2020-03-11T22:41:28.611 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pn": 0, 2020-03-11T22:41:28.611 INFO:tasks.workunit.client.0.smithi016.stdout: "accepted_pn": 0 2020-03-11T22:41:28.611 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:41:28.612 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:41:28.612 INFO:tasks.workunit.client.0.smithi016.stderr:report 927419779 2020-03-11T22:41:28.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:41:28.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-11T22:41:28.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:28.613 INFO:tasks.workunit.client.0.smithi016.stdout:delay recovery 2020-03-11T22:41:28.789 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:28.789 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.290383+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.790 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274644+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.790 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289456+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.790 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289693+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.790 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289878+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.790 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272548+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.790 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.222288+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.791 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.288961+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.791 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289007+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.791 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289233+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.791 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273910+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.791 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272802+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.791 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274300+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.792 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274552+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.792 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274831+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.792 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274550+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.792 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273364+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.792 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273669+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.792 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273246+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.793 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274123+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.793 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289135+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.793 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289037+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.793 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289518+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.795 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272568+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.795 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272564+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.796 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272542+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.796 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289654+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.796 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273250+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.796 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273249+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.797 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273854+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.797 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:32.273038+0000 44'101 44:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.797 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273942+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:28.798 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:41:28.798 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:28.798 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:41:28.799 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:28.800 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:41:28.803 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:41:28.803 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:41:28.803 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:28.803 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:28.804 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:41:28.805 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.0.asok 2020-03-11T22:41:28.805 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:28.805 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:41:28.897 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:28.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:28.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:41:28.906 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:41:28.906 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:41:28.906 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:28.906 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:28.907 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:41:28.907 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.2.asok 2020-03-11T22:41:28.907 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:28.907 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:41:29.002 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:29.009 INFO:tasks.workunit.client.0.smithi016.stdout:revive divergent 1 2020-03-11T22:41:29.010 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-11T22:41:29.010 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:29.354 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:29.354 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.290383+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.354 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274644+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.355 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289456+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.355 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289693+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.355 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289878+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.355 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272548+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.355 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.222288+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.355 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.288961+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.355 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289007+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289233+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273910+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272802+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274300+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274552+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274831+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.356 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274550+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.357 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273364+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.357 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273669+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.357 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273246+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.357 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274123+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.357 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289135+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.357 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289037+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.358 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289518+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.358 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272568+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.358 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272564+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.358 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272542+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.358 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289654+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.358 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273250+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.358 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273249+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.359 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273854+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.359 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:32.273038+0000 44'101 44:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.359 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273942+0000 30'100 44:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:29.359 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:41:29.359 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:29.360 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:41:29.364 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-11T22:41:30.153 INFO:tasks.workunit.client.0.smithi016.stderr:noup is set 2020-03-11T22:41:30.167 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:41:30.167 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:30.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:30.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:41:30.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:30.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:30.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:30.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:30.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:30.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:30.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:30.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:30.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:30.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:30.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:30.170 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:30.170 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:30.170 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:30.171 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:30.171 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:30.171 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:41:30.171 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:41:30.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:41:30.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:30.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:30.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:30.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:30.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:30.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:30.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:30.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:30.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:30.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:41:30.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:30.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:41:30.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:41:30.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:30.174 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:30.174 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:30.174 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:41:30.192 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:30.189+0000 7f2de465cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:30.193 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:30.191+0000 7f2de465cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:30.195 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:30.193+0000 7f2de465cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:30.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-11T22:41:30.711 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:30.709+0000 7f2de465cec0 -1 Falling back to public interface 2020-03-11T22:41:31.585 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:31.583+0000 7f2de465cec0 -1 osd.1 30 log_to_monitors {default=true} 2020-03-11T22:41:35.533 INFO:tasks.workunit.client.0.smithi016.stdout:delay recovery divergent 2020-03-11T22:41:35.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-11T22:41:35.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.290383+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274644+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.892 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289456+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289693+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289878+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272548+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.222288+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.288961+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.893 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289007+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289233+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273910+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272802+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274300+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274552+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274831+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274550+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273364+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273669+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273246+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.274123+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289135+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.895 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289037+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289518+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272568+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272564+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.272542+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.289654+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.896 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273250+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273249+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273854+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:32.273038+0000 44'101 45:134 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:32.273942+0000 30'100 45:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:41:35.897 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:35.898 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:41:35.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:41:35.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:41:35.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:41:35.903 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:35.903 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:35.903 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:41:35.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.1.asok 2020-03-11T22:41:35.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:35.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:41:35.999 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:36.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-11T22:41:36.800 INFO:tasks.workunit.client.0.smithi016.stderr:noup is unset 2020-03-11T22:41:36.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-11T22:41:36.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:36.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:41:36.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:36.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:36.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:36.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:36.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:36.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:36.811 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:41:37.214 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 36 up_thru 43 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/39346,v1:127.0.0.1:6803/39346] [v2:127.0.0.1:6804/39346,v1:127.0.0.1:6805/39346] exists,up 1adeefef-8ac6-40d7-ba02-a2dc9602cac5 2020-03-11T22:41:37.214 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:41:37.214 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:37.214 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:37.214 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:37.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:41:37.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:37.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:37.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:37.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:37.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:37.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:37.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:37.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:37.587 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42088,v1:127.0.0.1:6819/42088] [v2:127.0.0.1:6820/42088,v1:127.0.0.1:6821/42088] exists,up 55b263f4-40f2-4800-a15b-74333e0f0c12 2020-03-11T22:41:37.588 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:41:37.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:37.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:37.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:37.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-03-11T22:41:37.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:37.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:41:37.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:37.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:37.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:37.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:37.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:37.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:37.948 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 33 last_clean_interval [14,30) [v2:127.0.0.1:6810/39540,v1:127.0.0.1:6811/39540] [v2:127.0.0.1:6812/39540,v1:127.0.0.1:6813/39540] exists,up 50c76265-8eaa-44a7-8947-296ee0f56e9d 2020-03-11T22:41:37.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:37.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:37.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:37.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:38.288 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:38.289 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522770+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.289 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524154+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.289 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522917+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.290 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522960+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.290 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522833+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.290 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526490+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.290 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526864+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.290 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523589+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.291 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523575+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.291 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523465+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.291 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.525383+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.291 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526561+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.291 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524236+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.292 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524170+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.292 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524128+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.292 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524745+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.292 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526159+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.292 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:36.524878+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.292 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:36.525589+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.292 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:36.524783+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.293 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523538+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.293 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:36.524089+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.293 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523383+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.293 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526085+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.293 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526184+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.293 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526122+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.293 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522899+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.294 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.525884+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.294 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.525413+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.294 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524972+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.294 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.525499+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.294 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524797+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.294 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:41:38.294 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:38.295 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:41:38.303 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-03-11T22:41:38.304 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:38.304 INFO:tasks.workunit.client.0.smithi016.stdout:wait for peering 2020-03-11T22:41:38.641 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:38.641 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522770+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.642 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524154+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.642 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522917+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.642 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522960+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.642 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522833+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.642 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526490+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.642 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526864+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.642 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523589+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.643 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523575+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.643 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523465+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.643 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.525383+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.643 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526561+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.643 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524236+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.643 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524170+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.643 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524128+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.644 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524745+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.644 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526159+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.644 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:36.524878+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.644 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:36.525589+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.644 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:36.524783+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.644 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523538+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.645 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:36.524089+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.645 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.523383+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.645 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526085+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.645 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526184+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.645 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.526122+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.645 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.522899+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.645 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.525884+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.646 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.525413+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.646 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524972+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.646 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.525499+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.646 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:36.524797+0000 30'106 48:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:38.646 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:41:38.646 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:38.647 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:41:38.651 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-03-11T22:41:44.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-03-11T22:41:44.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:44.758 INFO:tasks.workunit.client.0.smithi016.stdout:killing divergent 1 2020-03-11T22:41:45.116 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:45.116 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.983986+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.116 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.982341+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.116 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.980352+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.116 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.985844+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.116 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:41:36.982619+0000 30'100 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.975865+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.984262+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.979958+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.984511+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.982496+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.980686+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.117 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.975869+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.118 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.979047+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.118 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.983915+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.118 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:41:36.982204+0000 30'100 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.118 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.980945+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.118 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.977870+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.118 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.978142+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.118 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.977289+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.119 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.982077+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.119 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.983033+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.119 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:41:36.982686+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.119 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.985478+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.119 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.983918+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.119 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.983668+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.119 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovery_wait+degraded+wait 2020-03-11T22:41:36.981844+0000 30'100 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.120 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.983778+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.120 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.977986+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.120 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:41:36.982912+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.120 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:41:36.983430+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.120 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovering+degraded+wait 2020-03-11T22:41:36.985507+0000 44'101 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.120 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.982086+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:45.120 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:41:45.120 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:45.121 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:41:45.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:41:45.125 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:45.126 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:45.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:45.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:45.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:45.231 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:45.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.17411.out 2020-03-11T22:41:45.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17411.out 2020-03-11T22:41:45.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:41:45.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:41:45.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:41:45.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:41:45.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:41:45.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17411.out 2020-03-11T22:41:45.786 INFO:tasks.workunit.client.0.smithi016.stderr:Exporting 1.0 info 1.0( v 30'100 (30'90,30'100] local-lis/les=48/49 n=3 ec=22/22 lis/c=48/48 les/c/f=49/49/0 sis=48 pruub=7.745403200s) 2020-03-11T22:41:45.798 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:41:45.811 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:41:45.828 INFO:tasks.workunit.client.0.smithi016.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:41:45.837 INFO:tasks.workunit.client.0.smithi016.stderr:Export successful 2020-03-11T22:41:45.838 INFO:tasks.workunit.client.0.smithi016.stdout: marking collection for removal 2020-03-11T22:41:45.838 INFO:tasks.workunit.client.0.smithi016.stdout:setting '_remove' omap key 2020-03-11T22:41:45.838 INFO:tasks.workunit.client.0.smithi016.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:41:45.879 INFO:tasks.workunit.client.0.smithi016.stderr:Remove successful 2020-03-11T22:41:46.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17411.out 2020-03-11T22:41:46.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:41:46.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:41:46.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:41:46.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:41:46.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:41:46.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17411.out 2020-03-11T22:41:46.838 INFO:tasks.workunit.client.0.smithi016.stdout:get_pg_num_history pg_num_history pg_num_history(e49 pg_nums {1={22=1,39=32}} deleted_pools ) 2020-03-11T22:41:46.838 INFO:tasks.workunit.client.0.smithi016.stdout:Importing pgid 1.0 2020-03-11T22:41:46.838 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:41:46.838 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:41:46.880 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:41:46.881 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:41:46.889 INFO:tasks.workunit.client.0.smithi016.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:41:46.889 INFO:tasks.workunit.client.0.smithi016.stdout:snapset 0=[]:{} 2020-03-11T22:41:46.897 INFO:tasks.workunit.client.0.smithi016.stdout:write_pg epoch 49 info 1.0( v 30'100 (30'90,30'100] local-lis/les=48/49 n=3 ec=22/22 lis/c=48/48 les/c/f=49/49/0 sis=48 pruub=7.745403200s) 2020-03-11T22:41:46.905 INFO:tasks.workunit.client.0.smithi016.stdout:Import successful 2020-03-11T22:41:47.424 INFO:tasks.workunit.client.0.smithi016.stdout:reviving divergent 1 2020-03-11T22:41:47.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-03-11T22:41:47.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:47.783 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:47.783 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.983986+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.784 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.982341+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.784 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.980352+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.784 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.985844+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.784 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:41:36.982619+0000 30'100 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.784 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.975865+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.784 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.984262+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.785 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.979958+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.785 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.984511+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.785 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.982496+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.785 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.980686+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.785 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.975869+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.786 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.979047+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.786 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.983915+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.786 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:41:36.982204+0000 30'100 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.786 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.980945+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.786 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.977870+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.786 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.978142+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.786 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.977289+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.787 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.982077+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.787 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.983033+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.787 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:41:36.982686+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.787 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.985478+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.787 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.983918+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.787 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.983668+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.788 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovery_wait+degraded+wait 2020-03-11T22:41:36.981844+0000 30'100 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.788 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.983778+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.788 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:36.977986+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.788 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:41:36.982912+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.788 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:41:36.983430+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.788 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovering+degraded+wait 2020-03-11T22:41:36.985507+0000 44'101 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.788 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:36.982086+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:41:47.788 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:41:47.789 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:47.789 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:41:47.796 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:41:47.796 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:47.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:47.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:41:47.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:47.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:47.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:47.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:47.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:47.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:47.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:47.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:47.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:47.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:47.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:47.799 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:47.799 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:47.799 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:47.799 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:47.799 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:47.799 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:41:47.799 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:41:47.799 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:41:47.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:47.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:47.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:47.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:47.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:47.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:47.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:47.801 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:47.801 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:47.801 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:41:47.801 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:47.801 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:41:47.801 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:41:47.802 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:41:47.802 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:47.802 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:47.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:47.824 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:47.821+0000 7f0758b5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:47.826 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:47.822+0000 7f0758b5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:47.828 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:47.824+0000 7f0758b5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:48.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:41:48.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:48.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:48.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:48.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:48.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:48.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:48.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:48.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:48.166 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:41:48.345 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:48.342+0000 7f0758b5fec0 -1 Falling back to public interface 2020-03-11T22:41:48.551 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42088,v1:127.0.0.1:6819/42088] [v2:127.0.0.1:6820/42088,v1:127.0.0.1:6821/42088] exists,up 55b263f4-40f2-4800-a15b-74333e0f0c12 2020-03-11T22:41:48.552 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:41:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:41:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:48.916 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42088,v1:127.0.0.1:6819/42088] [v2:127.0.0.1:6820/42088,v1:127.0.0.1:6821/42088] exists,up 55b263f4-40f2-4800-a15b-74333e0f0c12 2020-03-11T22:41:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-03-11T22:41:49.148 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:41:49.145+0000 7f0758b5fec0 -1 osd.1 49 log_to_monitors {default=true} 2020-03-11T22:42:08.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:42:08.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:42:08.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:42:08.919 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:08.919 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:08.919 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:08.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.1.asok 2020-03-11T22:42:08.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:42:08.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:42:09.022 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:42:09.022 INFO:tasks.workunit.client.0.smithi016.stdout: "ops": [], 2020-03-11T22:42:09.022 INFO:tasks.workunit.client.0.smithi016.stdout: "num_ops": 0 2020-03-11T22:42:09.022 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:42:09.030 INFO:tasks.workunit.client.0.smithi016.stdout:allowing recovery 2020-03-11T22:42:09.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-03-11T22:42:09.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:09.401 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:09.401 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:03.618734+0000 30'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.401 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:57.614862+0000 30'100 58:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.401 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.503922+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.401 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:50.107161+0000 30'100 59:146 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.401 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:42:03.619729+0000 30'100 59:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.402 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:57.674565+0000 30'100 58:150 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:41:54.160200+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.402 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.502555+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:41:52.213811+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.402 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:42:01.228378+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:01.228312+0000 30'100 2020-03-11T22:42:01.228312+0000 0 2020-03-11T22:42:09.402 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:03.620055+0000 30'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.402 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:57.675175+0000 30'100 58:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.402 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.492643+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.402 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:59.235100+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:41:59.235050+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.403 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:42:03.619560+0000 30'100 59:153 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:42:01.714286+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.403 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:42:05.468259+0000 30'100 59:153 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:42:05.468204+0000 30'100 2020-03-11T22:42:05.468204+0000 0 2020-03-11T22:42:09.403 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.491786+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.403 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:42:04.199542+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:04.199494+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.403 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:55.492543+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.403 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:56.214821+0000 30'100 59:149 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:41:56.214749+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.403 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:42:03.619745+0000 30'100 59:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.404 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:42:01.379032+0000 30'100 58:153 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:42:01.378981+0000 30'100 2020-03-11T22:42:01.378981+0000 0 2020-03-11T22:42:09.404 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:42:02.195948+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:02.195899+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.404 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:55.504106+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.404 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:41:57.674960+0000 49'101 58:145 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.404 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:42:03.619345+0000 30'100 59:154 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:41:53.209358+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.405 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:41:50.104203+0000 30'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.405 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:55.504255+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.405 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:57.673982+0000 30'100 58:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.405 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:42:03.606744+0000 30'100 59:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.405 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:42:08.123017+0000 30'100 60:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:08.122965+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.405 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.504107+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.406 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:41:57.674094+0000 44'101 58:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.406 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:03.619342+0000 30'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:09.406 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:42:09.406 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:09.406 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:42:09.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:42:09.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:42:09.591 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:09.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:42:09.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:42:09.729 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:09.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:42:09.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:42:09.864 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:09.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-03-11T22:42:09.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:09.872 INFO:tasks.workunit.client.0.smithi016.stdout:reading divergent objects 2020-03-11T22:42:10.212 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:10.213 INFO:tasks.workunit.client.0.smithi016.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:03.618734+0000 30'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.213 INFO:tasks.workunit.client.0.smithi016.stdout:1.1e 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:57.614862+0000 30'100 58:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.213 INFO:tasks.workunit.client.0.smithi016.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.503922+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.213 INFO:tasks.workunit.client.0.smithi016.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:50.107161+0000 30'100 59:146 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.213 INFO:tasks.workunit.client.0.smithi016.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:42:03.619729+0000 30'100 59:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.213 INFO:tasks.workunit.client.0.smithi016.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:57.674565+0000 30'100 58:150 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:41:54.160200+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.214 INFO:tasks.workunit.client.0.smithi016.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.502555+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:41:52.213811+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.214 INFO:tasks.workunit.client.0.smithi016.stdout:1.18 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:42:01.228378+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:01.228312+0000 30'100 2020-03-11T22:42:01.228312+0000 0 2020-03-11T22:42:10.214 INFO:tasks.workunit.client.0.smithi016.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:03.620055+0000 30'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.214 INFO:tasks.workunit.client.0.smithi016.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:57.675175+0000 30'100 58:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.214 INFO:tasks.workunit.client.0.smithi016.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.492643+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.214 INFO:tasks.workunit.client.0.smithi016.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:59.235100+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:41:59.235050+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.214 INFO:tasks.workunit.client.0.smithi016.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:42:03.619560+0000 30'100 59:153 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:42:01.714286+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.215 INFO:tasks.workunit.client.0.smithi016.stdout:1.12 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:42:05.468259+0000 30'100 59:153 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:42:05.468204+0000 30'100 2020-03-11T22:42:05.468204+0000 0 2020-03-11T22:42:10.215 INFO:tasks.workunit.client.0.smithi016.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.491786+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.215 INFO:tasks.workunit.client.0.smithi016.stdout:1.10 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:42:04.199542+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:04.199494+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.215 INFO:tasks.workunit.client.0.smithi016.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:55.492543+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.215 INFO:tasks.workunit.client.0.smithi016.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:56.214821+0000 30'100 59:149 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:41:56.214749+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.215 INFO:tasks.workunit.client.0.smithi016.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:42:03.619745+0000 30'100 59:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.215 INFO:tasks.workunit.client.0.smithi016.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:42:01.379032+0000 30'100 58:153 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:42:01.378981+0000 30'100 2020-03-11T22:42:01.378981+0000 0 2020-03-11T22:42:10.215 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:42:02.195948+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:02.195899+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.216 INFO:tasks.workunit.client.0.smithi016.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:55.504106+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.216 INFO:tasks.workunit.client.0.smithi016.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:41:57.674960+0000 49'101 58:145 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.216 INFO:tasks.workunit.client.0.smithi016.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:42:03.619345+0000 30'100 59:154 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:41:53.209358+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.216 INFO:tasks.workunit.client.0.smithi016.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:41:50.104203+0000 30'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.216 INFO:tasks.workunit.client.0.smithi016.stdout:1.9 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:55.504255+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.216 INFO:tasks.workunit.client.0.smithi016.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:57.673982+0000 30'100 58:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.216 INFO:tasks.workunit.client.0.smithi016.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:42:03.606744+0000 30'100 59:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.217 INFO:tasks.workunit.client.0.smithi016.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:42:08.123017+0000 30'100 60:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:08.122965+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.217 INFO:tasks.workunit.client.0.smithi016.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:55.504107+0000 30'100 57:127 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.217 INFO:tasks.workunit.client.0.smithi016.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:41:57.674094+0000 44'101 58:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.217 INFO:tasks.workunit.client.0.smithi016.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:03.619342+0000 30'100 59:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:31.752858+0000 0'0 2020-03-11T22:39:31.752858+0000 0 2020-03-11T22:42:10.217 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:42:10.217 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:10.218 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:42:10.222 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-03-11T22:42:10.223 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-03-11T22:42:10.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:10.225 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:42:10.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:10.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:42:10.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:10.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:42:10.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:10.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:42:10.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-03-11T22:42:10.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:42:10.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:42:10.374 INFO:tasks.workunit.client.0.smithi016.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:42:10.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:42:10.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:42:10.403 INFO:tasks.workunit.client.0.smithi016.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:42:10.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-03-11T22:42:10.406 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:10.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:10.416 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4542.0:1 2020-03-11T22:39:57.251395+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.416 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 20 _merge_object_divergent_entries: keeping 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4542.0:1 2020-03-11T22:39:57.251395+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.417 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:10.417 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 30'2 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-03-11T22:42:10.417 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:42:10.417 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:10.417 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 30'106 2020-03-11T22:42:10.417 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4542.0:1 2020-03-11T22:39:57.251395+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.418 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:42:10.418 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.639+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 30'2 <= info.log_tail 30'89 2020-03-11T22:42:10.418 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 30'106 (30'4) delete 1:70b3bf12:::existing_4:head by client.4545.0:1 2020-03-11T22:39:57.256049+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:10.418 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 20 _merge_object_divergent_entries: keeping 30'106 (30'4) delete 1:70b3bf12:::existing_4:head by client.4545.0:1 2020-03-11T22:39:57.256049+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:10.418 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:10.418 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 30'4 first_divergent_update: 30'106 last_divergent_update: 30'106 2020-03-11T22:42:10.418 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 10 _merge_object_divergent_entries: more recent entry found: 44'101 (30'4) modify 1:70b3bf12:::existing_4:head by client.4656.0:1 2020-03-11T22:40:52.634595+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:42:10.419 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 30'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4537.0:1 2020-03-11T22:39:57.246452+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.419 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 20 _merge_object_divergent_entries: keeping 30'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4537.0:1 2020-03-11T22:39:57.246452+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.420 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:10.420 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-03-11T22:42:10.420 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:42:10.420 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.714+0000 7f2dc0813700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:42:10.420 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:39:57.253359+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:10.420 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 20 _merge_object_divergent_entries: keeping 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:39:57.253359+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:10.420 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:42:10.421 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 30'105 last_divergent_update: 30'105 2020-03-11T22:42:10.421 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:42:10.421 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:10.421 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 30'106 2020-03-11T22:42:10.421 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:39:57.253359+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:10.421 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:42:10.422 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.715+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'84 2020-03-11T22:42:10.422 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4533.0:1 2020-03-11T22:39:57.238275+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.422 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 20 _merge_object_divergent_entries: keeping 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4533.0:1 2020-03-11T22:39:57.238275+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.422 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:10.424 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 30'1 first_divergent_update: 30'102 last_divergent_update: 30'102 2020-03-11T22:42:10.424 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:42:10.424 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:10.424 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 30'106 2020-03-11T22:42:10.424 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4533.0:1 2020-03-11T22:39:57.238275+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.425 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:42:10.425 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 30'1 <= info.log_tail 30'90 2020-03-11T22:42:10.425 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 30'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-03-11T22:39:57.231411+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.425 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 20 _merge_object_divergent_entries: keeping 30'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-03-11T22:39:57.231411+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:10.425 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:10.425 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 30'101 last_divergent_update: 30'101 2020-03-11T22:42:10.426 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:42:10.426 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:36.716+0000 7f2dc1815700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:42:10.426 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:10.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:10.434 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:10.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:10.444 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-03-11T22:42:10.444 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.17411.out 2020-03-11T22:42:10.444 INFO:tasks.workunit.client.0.smithi016.stdout:success 2020-03-11T22:42:10.445 INFO:tasks.workunit.client.0.smithi016.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17411.out': No such file or directory 2020-03-11T22:42:10.445 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-03-11T22:42:10.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:42:10.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:42:10.977 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' does not exist 2020-03-11T22:42:10.985 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:11.094 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:11.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:42:11.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:42:11.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:42:11.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:42:11.095 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:11.095 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:11.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:11.096 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:11.096 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:11.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:11.100 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:42:11.100 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:42:11.101 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:42:11.101 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:42:11.102 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:42:11.102 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:42:11.103 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:11.104 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:42:11.104 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:42:11.104 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:11.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:42:11.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:42:11.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:42:11.123 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:42:11.123 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:11.124 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:11.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:42:11.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:42:11.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:42:11.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:42:11.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:42:11.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:42:11.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:42:11.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:42:11.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:42:11.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:42:11.127 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:11.127 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:11.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:11.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:11.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:11.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:11.129 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:42:11.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:42:11.129 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:42:11.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:42:11.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:42:11.130 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:42:11.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:42:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:42:11.131 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:11.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:42:11.133 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:42:11.133 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:42:11.134 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:42:11.134 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:11.134 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:11.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:42:11.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:42:11.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:42:11.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:42:11.136 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:42:11.136 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:11.136 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:11.136 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17411 2020-03-11T22:42:11.138 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:42:11.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:42:11.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:42:11.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:42:11.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-03-11T22:42:11.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-03-11T22:42:11.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-03-11T22:42:11.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-03-11T22:42:11.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-03-11T22:42:11.140 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-03-11T22:42:11.141 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-03-11T22:42:11.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-03-11T22:42:11.141 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:42:11.141 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:42:11.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-03-11T22:42:11.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:42:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:42:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:42:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:42:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:42:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:42:11.407 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:42:11.407 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:11.407 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:11.407 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:11.407 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:11.408 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:11.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:42:11.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:42:11.549 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:42:11.551 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:42:11.551 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:42:11.551 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:42:11.551 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:42:11.551 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:42:11.551 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:42:11.551 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:42:11.552 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:42:11.552 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:11.552 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:11.552 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:11.553 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-mon.a.asok 2020-03-11T22:42:11.553 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:42:11.553 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17411/ceph-mon.a.asok config get fsid 2020-03-11T22:42:11.643 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:42:11.644 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:42:11.644 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:42:11.644 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:42:11.644 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:42:11.644 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:42:11.644 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:42:11.644 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:42:11.645 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:11.645 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:11.645 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:11.645 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-mon.a.asok 2020-03-11T22:42:11.645 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:42:11.646 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17411/ceph-mon.a.asok config get mon_host 2020-03-11T22:42:11.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-03-11T22:42:11.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:42:11.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:42:11.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:42:11.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:42:11.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:42:11.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:42:12.636 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:42:12.637 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:12.637 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:12.637 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:12.637 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:12.637 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:12.638 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:42:12.638 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:42:12.638 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:42:12.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:12.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-03-11T22:42:12.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:12.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:12.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:42:12.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:12.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:42:12.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:12.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:12.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:12.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:12.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:42:12.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:42:12.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:12.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:12.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:12.670 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:12.670 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:12.670 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:12.671 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:12.671 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:12.671 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:12.671 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:42:12.671 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:42:12.671 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:12.671 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:12.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:12.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:12.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:12.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:12.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:12.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:12.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:12.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:12.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:42:12.673 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:12.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6c73bd18-95db-4897-9cb7-6ae241013376 2020-03-11T22:42:12.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 6c73bd18-95db-4897-9cb7-6ae241013376' 2020-03-11T22:42:12.674 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:12.674 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 6c73bd18-95db-4897-9cb7-6ae241013376 2020-03-11T22:42:12.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBEaWle6tHdKBAAVfQUsRHorJvb2MovBFKIHw== 2020-03-11T22:42:12.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBEaWle6tHdKBAAVfQUsRHorJvb2MovBFKIHw=="}' 2020-03-11T22:42:12.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6c73bd18-95db-4897-9cb7-6ae241013376 -i td/divergent-priors/0/new.json 2020-03-11T22:42:13.051 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:42:13.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:42:13.062 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBEaWle6tHdKBAAVfQUsRHorJvb2MovBFKIHw== --osd-uuid 6c73bd18-95db-4897-9cb7-6ae241013376 2020-03-11T22:42:13.092 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:13.088+0000 7f86c5564ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:13.093 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:13.090+0000 7f86c5564ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:13.095 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:13.091+0000 7f86c5564ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:13.096 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:13.092+0000 7f86c5564ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:13.096 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:13.092+0000 7f86c5564ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:13.096 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:13.092+0000 7f86c5564ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:13.149 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:13.145+0000 7f86c5564ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:42:16.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:42:16.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:16.202 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:42:16.203 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:42:16.203 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:16.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:42:16.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:16.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:16.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:16.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:16.683 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:42:16.709 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:16.706+0000 7fbba3462ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:16.711 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:16.707+0000 7fbba3462ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:16.712 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:16.709+0000 7fbba3462ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:17.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:42:17.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:17.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:42:17.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:17.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:17.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:17.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:17.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:17.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:17.050 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:42:17.229 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:17.225+0000 7fbba3462ec0 -1 Falling back to public interface 2020-03-11T22:42:17.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:17.982 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:17.978+0000 7fbba3462ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:42:18.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:18.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:18.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:18.489 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:42:18.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:18.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:18.880 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/43589,v1:127.0.0.1:6803/43589] [v2:127.0.0.1:6804/43589,v1:127.0.0.1:6805/43589] exists,up 6c73bd18-95db-4897-9cb7-6ae241013376 2020-03-11T22:42:18.880 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 6e1b8749-4a8a-4a6e-8118-60aab81e6f12 2020-03-11T22:42:18.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:18.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:18.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:18.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:18.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-03-11T22:42:18.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:18.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:18.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:42:18.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:18.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:42:18.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:18.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:18.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:18.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:18.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:42:18.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:42:18.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:18.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:18.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:18.883 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:18.883 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:18.884 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:18.884 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:18.884 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:18.884 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:18.884 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:42:18.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:42:18.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:18.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:18.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:18.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:18.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:18.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:18.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:18.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:18.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:18.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:18.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:42:18.887 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:18.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6e1b8749-4a8a-4a6e-8118-60aab81e6f12 2020-03-11T22:42:18.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 6e1b8749-4a8a-4a6e-8118-60aab81e6f12' 2020-03-11T22:42:18.887 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:18.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBKaWleXuNqNRAAeqR5zfbkvCTRx8XO/HSifA== 2020-03-11T22:42:18.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBKaWleXuNqNRAAeqR5zfbkvCTRx8XO/HSifA=="}' 2020-03-11T22:42:18.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6e1b8749-4a8a-4a6e-8118-60aab81e6f12 -i td/divergent-priors/1/new.json 2020-03-11T22:42:19.413 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:42:19.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:42:19.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBKaWleXuNqNRAAeqR5zfbkvCTRx8XO/HSifA== --osd-uuid 6e1b8749-4a8a-4a6e-8118-60aab81e6f12 2020-03-11T22:42:19.454 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:19.451+0000 7f5cbdba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:19.464 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:19.460+0000 7f5cbdba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:19.464 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:19.461+0000 7f5cbdba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:19.465 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:19.461+0000 7f5cbdba1ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:19.465 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:19.461+0000 7f5cbdba1ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:19.466 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:19.461+0000 7f5cbdba1ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:19.531 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:19.528+0000 7f5cbdba1ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:42:22.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:42:22.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:22.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:42:22.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:22.608 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:42:23.124 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:42:23.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:42:23.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:23.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:23.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:23.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:23.152 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:23.148+0000 7fbacedfaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:23.153 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:23.150+0000 7fbacedfaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:23.154 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:23.151+0000 7fbacedfaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:23.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:42:23.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:23.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:23.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:23.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:23.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:23.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:23.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:23.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:23.519 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:42:23.670 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:23.667+0000 7fbacedfaec0 -1 Falling back to public interface 2020-03-11T22:42:23.878 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:24.413 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:24.410+0000 7fbacedfaec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:42:24.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:24.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:24.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:24.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:24.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:24.881 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:42:25.279 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43944,v1:127.0.0.1:6811/43944] [v2:127.0.0.1:6812/43944,v1:127.0.0.1:6813/43944] exists,up 6e1b8749-4a8a-4a6e-8118-60aab81e6f12 2020-03-11T22:42:25.280 INFO:tasks.workunit.client.0.smithi016.stdout:add osd2 6d353a96-21d2-42eb-bd15-df39bc2f3fbb 2020-03-11T22:42:25.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:25.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:25.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:25.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:25.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-03-11T22:42:25.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:25.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:25.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:42:25.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:25.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:42:25.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:25.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:25.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:25.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:25.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:42:25.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:42:25.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:25.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:25.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:25.283 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:25.283 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:25.283 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:25.283 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:25.283 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:25.284 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:42:25.284 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:42:25.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:42:25.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:25.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:25.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:25.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:25.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:25.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:25.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:25.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:25.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:25.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:25.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:42:25.286 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:25.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6d353a96-21d2-42eb-bd15-df39bc2f3fbb 2020-03-11T22:42:25.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 6d353a96-21d2-42eb-bd15-df39bc2f3fbb' 2020-03-11T22:42:25.286 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:25.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBRaWleuWmwERAAQmUCLl0HlNQxACSUrtxPIw== 2020-03-11T22:42:25.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBRaWleuWmwERAAQmUCLl0HlNQxACSUrtxPIw=="}' 2020-03-11T22:42:25.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6d353a96-21d2-42eb-bd15-df39bc2f3fbb -i td/divergent-priors/2/new.json 2020-03-11T22:42:25.781 INFO:tasks.workunit.client.0.smithi016.stdout:2 2020-03-11T22:42:25.791 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:42:25.792 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBRaWleuWmwERAAQmUCLl0HlNQxACSUrtxPIw== --osd-uuid 6d353a96-21d2-42eb-bd15-df39bc2f3fbb 2020-03-11T22:42:25.822 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:25.818+0000 7f127c882ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:25.824 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:25.820+0000 7f127c882ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:25.825 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:25.822+0000 7f127c882ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:25.826 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:25.822+0000 7f127c882ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:25.826 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:25.823+0000 7f127c882ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:25.826 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:25.823+0000 7f127c882ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:25.878 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:25.876+0000 7f127c882ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:42:29.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:42:29.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:29.072 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd2 key to auth repository 2020-03-11T22:42:29.073 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:42:29.073 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:29.557 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:42:29.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:42:29.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:29.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:29.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:29.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:29.585 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:29.582+0000 7f91d3ed8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:29.586 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:29.584+0000 7f91d3ed8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:29.588 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:29.586+0000 7f91d3ed8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:29.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:29.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:29.926 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:29.926 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:29.926 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:29.926 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:42:30.105 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:30.102+0000 7f91d3ed8ec0 -1 Falling back to public interface 2020-03-11T22:42:30.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:30.805 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:42:30.802+0000 7f91d3ed8ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:42:31.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:31.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:31.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:31.294 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:42:31.295 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:31.295 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:31.690 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44299,v1:127.0.0.1:6819/44299] [v2:127.0.0.1:6820/44299,v1:127.0.0.1:6821/44299] exists,up 6d353a96-21d2-42eb-bd15-df39bc2f3fbb 2020-03-11T22:42:31.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:31.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:31.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:31.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-03-11T22:42:32.380 INFO:tasks.workunit.client.0.smithi016.stderr:noout is set 2020-03-11T22:42:32.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-03-11T22:42:33.233 INFO:tasks.workunit.client.0.smithi016.stderr:noin is set 2020-03-11T22:42:33.241 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-03-11T22:42:33.923 INFO:tasks.workunit.client.0.smithi016.stderr:nodown is set 2020-03-11T22:42:33.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-03-11T22:42:33.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: create_ec_pool: local pool_name=test 2020-03-11T22:42:33.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: create_ec_pool: shift 2020-03-11T22:42:33.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: create_ec_pool: local allow_overwrites=true 2020-03-11T22:42:33.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: create_ec_pool: shift 2020-03-11T22:42:33.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-03-11T22:42:34.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-03-11T22:42:34.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-03-11T22:42:35.158 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' already exists 2020-03-11T22:42:35.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:42:36.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: '[' true = true ']' 2020-03-11T22:42:36.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-03-11T22:42:36.899 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 allow_ec_overwrites to true 2020-03-11T22:42:36.908 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: wait_for_clean 2020-03-11T22:42:36.908 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:36.908 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:36.908 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:36.908 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:36.909 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:42:36.909 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:36.909 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:36.911 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:36.912 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:36.912 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:37.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:37.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:37.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:37.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:37.005 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:37.370 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:37.371 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:42:37.371 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:42:37.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:37.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:37.371 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:37.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:42:37.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:42:37.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:37.563 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:37.698 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:42:37.698 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:42:37.699 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:37.699 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:37.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:42:37.835 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509444' 2020-03-11T22:42:37.835 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:37.835 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:42:37.836 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:37.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:37.836 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:42:37.837 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:37.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:42:37.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:42:37.838 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:37.838 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:42:38.223 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:42:38.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:39.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:39.225 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:39.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:42:39.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:39.601 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:42:39.601 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:39.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:39.602 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:42:39.602 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:39.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:42:39.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:42:39.603 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:42:39.604 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:40.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:42:40.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:40.006 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:42:40.006 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:40.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:40.007 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:42:40.007 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:40.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:42:40.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:42:40.008 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:42:40.009 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:40.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:42:40.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:40.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:40.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:40.844 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:40.845 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:40.845 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:40.845 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:40.845 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:40.845 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:40.845 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:40.846 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:41.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:41.193 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:41.194 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:41.194 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:41.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:41.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:41.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:41.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: return 0 2020-03-11T22:42:41.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-03-11T22:42:41.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:41.669 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:42.036 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:42.036 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:42.229 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:42:42.230 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:42:42.230 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:42.230 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:42.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:42:42.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672969' 2020-03-11T22:42:42.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:42.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:42.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509447 2020-03-11T22:42:42.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-64424509447' 2020-03-11T22:42:42.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:42.503 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:42:42.504 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:42.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:42.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:42:42.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:42.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:42:42.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:42:42.507 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:42.507 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:42:42.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:42:42.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:43.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:43.871 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:44.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:42:44.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:44.263 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:42:44.264 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:44.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:44.265 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:42:44.265 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:44.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:42:44.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:42:44.266 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:44.267 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:42:44.705 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672969 2020-03-11T22:42:44.705 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:44.706 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509447 2020-03-11T22:42:44.706 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:44.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:44.707 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509447 2020-03-11T22:42:44.707 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:44.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509447 2020-03-11T22:42:44.709 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2020-03-11T22:42:44.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:44.709 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509447 2020-03-11T22:42:45.073 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509447 2020-03-11T22:42:45.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-03-11T22:42:45.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:45.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:45.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:45.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:45.074 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:42:45.074 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:45.075 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:45.075 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:45.075 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:45.075 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:45.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:45.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:45.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:45.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:45.171 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:45.549 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:45.549 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:42:45.550 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:42:45.550 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:45.550 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:45.550 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:45.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:42:45.684 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:42:45.684 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:45.684 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:45.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:42:45.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672972' 2020-03-11T22:42:45.820 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:45.820 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:45.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-03-11T22:42:45.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509449' 2020-03-11T22:42:45.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:45.957 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:42:45.957 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:45.958 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:45.958 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:42:45.958 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:45.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:42:45.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:42:45.960 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:45.960 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:42:46.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:42:46.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:46.324 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:42:46.324 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:46.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:46.325 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:42:46.326 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:46.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:42:46.327 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:42:46.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:46.327 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:42:46.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:42:46.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:46.716 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509449 2020-03-11T22:42:46.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:46.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:46.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509449 2020-03-11T22:42:46.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:46.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509449 2020-03-11T22:42:46.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-03-11T22:42:46.720 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:46.720 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509449 2020-03-11T22:42:47.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509449 -lt 64424509449 2020-03-11T22:42:47.118 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:47.118 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:47.118 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:47.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:47.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:47.588 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:47.588 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:47.588 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:47.589 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:47.589 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:47.589 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:47.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:47.938 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:47.938 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:47.938 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:48.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:48.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:48.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:48.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-03-11T22:42:48.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-03-11T22:42:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:42:48.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-03-11T22:42:48.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-03-11T22:42:48.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:42:48.800 INFO:tasks.workunit.client.0.smithi016.stdout:primary and soon to be divergent is 1 2020-03-11T22:42:49.139 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:49.139 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:42:36.083549+0000 0'0 28:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:42:49.139 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:42:49.139 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:49.140 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:42:49.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-03-11T22:42:49.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:49.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-03-11T22:42:49.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-03-11T22:42:49.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:49.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-03-11T22:42:49.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-03-11T22:42:49.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:49.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-03-11T22:42:49.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-03-11T22:42:49.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-03-11T22:42:49.151 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-03-11T22:42:49.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-03-11T22:42:49.152 INFO:tasks.workunit.client.0.smithi016.stdout:writing initial objects 2020-03-11T22:42:49.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-03-11T22:42:49.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-03-11T22:42:49.373 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.374 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-03-11T22:42:49.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-03-11T22:42:49.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-03-11T22:42:49.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-03-11T22:42:49.675 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-03-11T22:42:49.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-03-11T22:42:49.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-03-11T22:42:49.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-03-11T22:42:49.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-03-11T22:42:50.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-03-11T22:42:50.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-03-11T22:42:50.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.120 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-03-11T22:42:50.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-03-11T22:42:50.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-03-11T22:42:50.393 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-03-11T22:42:50.540 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.540 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-03-11T22:42:50.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-03-11T22:42:50.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-03-11T22:42:50.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-03-11T22:42:50.811 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.811 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-03-11T22:42:50.869 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.869 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-03-11T22:42:50.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-03-11T22:42:50.969 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.969 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-03-11T22:42:51.090 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.091 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-03-11T22:42:51.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-03-11T22:42:51.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.236 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-03-11T22:42:51.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-03-11T22:42:51.336 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.336 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-03-11T22:42:51.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.462 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-03-11T22:42:51.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-03-11T22:42:51.570 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.570 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-03-11T22:42:51.619 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.619 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-03-11T22:42:51.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.737 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-03-11T22:42:51.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-03-11T22:42:52.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-03-11T22:42:52.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-03-11T22:42:52.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.112 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-03-11T22:42:52.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-03-11T22:42:52.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-03-11T22:42:52.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-03-11T22:42:52.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-03-11T22:42:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.570 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-03-11T22:42:52.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-03-11T22:42:52.740 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.741 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-03-11T22:42:52.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-03-11T22:42:52.912 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.912 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-03-11T22:42:52.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-03-11T22:42:53.012 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.012 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-03-11T22:42:53.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-03-11T22:42:53.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-03-11T22:42:53.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-03-11T22:42:53.354 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.354 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-03-11T22:42:53.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-03-11T22:42:53.540 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.541 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-03-11T22:42:53.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-03-11T22:42:53.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-03-11T22:42:53.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-03-11T22:42:53.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-03-11T22:42:53.891 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.891 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-03-11T22:42:53.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-03-11T22:42:54.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.038 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-03-11T22:42:54.086 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.086 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-03-11T22:42:54.241 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.242 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-03-11T22:42:54.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-03-11T22:42:54.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-03-11T22:42:54.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-03-11T22:42:54.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-03-11T22:42:54.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-03-11T22:42:54.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-03-11T22:42:54.829 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-03-11T22:42:54.878 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.878 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-03-11T22:42:54.930 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.930 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-03-11T22:42:55.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-03-11T22:42:55.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-03-11T22:42:55.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-03-11T22:42:55.258 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-03-11T22:42:55.313 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.313 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-03-11T22:42:55.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-03-11T22:42:55.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-03-11T22:42:55.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-03-11T22:42:55.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-03-11T22:42:55.637 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.637 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-03-11T22:42:55.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-03-11T22:42:55.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-03-11T22:42:55.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-03-11T22:42:56.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-03-11T22:42:56.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.087 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-03-11T22:42:56.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-03-11T22:42:56.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-03-11T22:42:56.322 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.322 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-03-11T22:42:56.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-03-11T22:42:56.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-03-11T22:42:56.587 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.587 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-03-11T22:42:56.671 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.671 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-03-11T22:42:56.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-03-11T22:42:56.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:42:56.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.946 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:42:56.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:42:56.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-03-11T22:42:56.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:56.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:56.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:56.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:56.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:42:56.998 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:57.001 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:57.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:57.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:57.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:57.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:57.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:57.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:57.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:57.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:57.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:57.426 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:42:57.426 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:42:57.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:57.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:57.426 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:57.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836496 2020-03-11T22:42:57.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496' 2020-03-11T22:42:57.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:57.581 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:57.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672976 2020-03-11T22:42:57.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672976' 2020-03-11T22:42:57.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:57.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:57.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509454 2020-03-11T22:42:57.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672976 2-64424509454' 2020-03-11T22:42:57.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:57.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836496 2020-03-11T22:42:57.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:57.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:57.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836496 2020-03-11T22:42:57.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:57.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836496 2020-03-11T22:42:57.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-03-11T22:42:57.857 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:57.857 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836496 2020-03-11T22:42:58.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836496 2020-03-11T22:42:58.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:59.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:59.240 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:59.618 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836496 -lt 21474836496 2020-03-11T22:42:59.619 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:59.619 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672976 2020-03-11T22:42:59.619 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:59.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:59.620 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672976 2020-03-11T22:42:59.620 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:59.621 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672976 2020-03-11T22:42:59.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2020-03-11T22:42:59.622 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:59.622 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672976 2020-03-11T22:43:00.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 42949672976 2020-03-11T22:43:00.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:00.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509454 2020-03-11T22:43:00.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:00.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:00.024 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509454 2020-03-11T22:43:00.024 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:00.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509454 2020-03-11T22:43:00.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2020-03-11T22:43:00.025 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:00.026 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509454 2020-03-11T22:43:00.406 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509454 -lt 64424509454 2020-03-11T22:43:00.407 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:00.407 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:00.407 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:00.879 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:00.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:00.880 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:00.880 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:00.880 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:00.880 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:00.880 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:00.880 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:01.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:43:01.235 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:01.235 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:01.235 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:01.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:43:01.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:43:01.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:43:01.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-03-11T22:43:01.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:43:01.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:43:01.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:43:01.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi016.stdout:blackholing osds 0 2 2020-03-11T22:43:02.421 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:02.421 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:42:36.083549+0000 29'100 29:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:43:02.421 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:43:02.421 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:02.422 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:43:02.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:02.434 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:43:02.434 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:43:02.434 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:43:02.434 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:02.435 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:02.435 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:43:02.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.0.asok 2020-03-11T22:43:02.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:43:02.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:43:02.523 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:43:02.523 INFO:tasks.workunit.client.0.smithi016.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:43:02.523 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:43:02.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:02.531 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:43:02.531 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:43:02.531 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:43:02.532 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:02.532 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:02.532 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:43:02.532 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.2.asok 2020-03-11T22:43:02.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:43:02.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:43:02.623 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:43:02.623 INFO:tasks.workunit.client.0.smithi016.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:43:02.623 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:43:02.630 INFO:tasks.workunit.client.0.smithi016.stdout:writing divergent object 2020-03-11T22:43:02.631 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-03-11T22:43:02.631 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-03-11T22:43:02.631 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-03-11T22:43:03.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-03-11T22:43:03.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:43:03.817 INFO:tasks.workunit.client.0.smithi016.stdout:created pool test snap snap1 2020-03-11T22:43:03.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-03-11T22:43:03.819 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-03-11T22:43:03.820 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:43:13.820 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-03-11T22:43:13.825 INFO:tasks.workunit.client.0.smithi016.stderr:rados: no process found 2020-03-11T22:43:13.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-03-11T22:43:13.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:13.826 INFO:tasks.workunit.client.0.smithi016.stdout:killing all the osds 2020-03-11T22:43:14.175 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:14.175 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2020-03-11T22:42:36.083549+0000 30'104 30:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:43:14.175 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:43:14.175 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:14.176 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:43:14.184 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:43:14.185 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:14.185 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:14.185 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:14.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:14.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:14.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:14.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:14.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-03-11T22:43:14.859 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already down. 2020-03-11T22:43:14.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:14.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-03-11T22:43:15.445 INFO:tasks.workunit.client.0.smithi016.stderr:osd.1 is already down. 2020-03-11T22:43:15.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:15.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-03-11T22:43:15.995 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already down. 2020-03-11T22:43:16.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:16.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-03-11T22:43:16.620 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already out. 2020-03-11T22:43:16.628 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:16.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-03-11T22:43:17.162 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already out. 2020-03-11T22:43:17.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-03-11T22:43:17.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:17.171 INFO:tasks.workunit.client.0.smithi016.stdout:bringing up non_divergent 0 2 2020-03-11T22:43:17.508 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:17.508 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2020-03-11T22:42:36.083549+0000 30'104 30:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:43:17.509 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:43:17.509 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:17.510 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:43:17.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:17.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:43:17.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:17.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:17.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:43:17.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:17.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:17.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:17.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:17.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:17.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:17.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:17.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:17.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:17.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:17.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:17.525 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:17.525 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:17.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:17.526 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:17.526 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:17.526 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:43:17.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:43:17.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:43:17.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:17.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:17.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:17.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:17.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:17.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:17.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:17.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:17.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:17.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:43:17.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:17.529 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:43:17.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:43:17.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:17.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:17.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:17.530 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:43:17.549 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:17.545+0000 7f2c0d354ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:17.550 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:17.547+0000 7f2c0d354ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:17.552 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:17.549+0000 7f2c0d354ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:17.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:43:17.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:17.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:17.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:17.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:17.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:17.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:17.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:17.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:17.888 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:43:18.070 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:18.066+0000 7f2c0d354ec0 -1 Falling back to public interface 2020-03-11T22:43:18.289 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:18.924 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:18.921+0000 7f2c0d354ec0 -1 osd.0 29 log_to_monitors {default=true} 2020-03-11T22:43:19.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:19.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:19.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:19.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:19.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:19.292 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:43:19.671 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/48703,v1:127.0.0.1:6803/48703] [v2:127.0.0.1:6804/48703,v1:127.0.0.1:6805/48703] exists,up 6c73bd18-95db-4897-9cb7-6ae241013376 2020-03-11T22:43:19.671 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:43:19.671 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:19.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:19.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:19.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:19.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:43:19.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:19.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:19.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:43:19.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:19.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:19.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:19.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:19.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:19.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:19.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:19.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:19.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:19.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:19.675 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:19.675 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:19.675 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:19.675 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:19.675 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:19.676 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:19.676 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:43:19.676 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:43:19.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:43:19.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:19.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:19.677 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:19.677 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:19.677 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:19.677 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:19.677 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:19.677 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:19.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:19.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:43:19.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:19.678 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:43:19.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:43:19.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:19.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:19.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:19.697 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:19.693+0000 7fd23f779ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:19.699 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:19.695+0000 7fd23f779ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:19.700 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:19.697+0000 7fd23f779ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:20.038 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:43:20.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:20.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:20.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:20.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:20.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:20.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:20.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:20.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:20.040 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:43:20.217 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:20.214+0000 7fd23f779ec0 -1 Falling back to public interface 2020-03-11T22:43:20.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:21.086 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:43:21.083+0000 7fd23f779ec0 -1 osd.2 29 log_to_monitors {default=true} 2020-03-11T22:43:21.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:21.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:21.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:21.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:21.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:21.448 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:43:21.837 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/48897,v1:127.0.0.1:6811/48897] [v2:127.0.0.1:6812/48897,v1:127.0.0.1:6813/48897] exists,up 6d353a96-21d2-42eb-bd15-df39bc2f3fbb 2020-03-11T22:43:21.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:21.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:21.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:21.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:21.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-03-11T22:43:22.380 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already in. 2020-03-11T22:43:22.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:22.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-03-11T22:43:22.982 INFO:tasks.workunit.client.0.smithi016.stderr:osd.2 is already in. 2020-03-11T22:43:22.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-03-11T22:43:27.992 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:43:27.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-03-11T22:43:27.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-03-11T22:43:27.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:27.993 INFO:tasks.workunit.client.0.smithi016.stdout:writing non-divergent object existing_4 2020-03-11T22:43:28.345 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:28.346 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:23.105154+0000 29'100 40:120 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:43:28.346 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:43:28.346 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:28.347 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:43:28.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:43:28.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:43:28.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-03-11T22:43:28.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:28.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:28.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:28.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:28.412 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:43:28.413 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:28.413 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:28.414 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:28.414 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:28.414 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:28.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:28.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:28.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:28.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:28.473 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:28.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:28.838 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:43:28.838 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:43:28.839 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:28.839 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:28.839 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:29.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822661 2020-03-11T22:43:29.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822661' 2020-03-11T22:43:29.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:29.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:29.103 INFO:tasks.workunit.client.0.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:43:29.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:43:29.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822661 1-' 2020-03-11T22:43:29.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:29.112 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:29.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789956 2020-03-11T22:43:29.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822661 1- 2-158913789956' 2020-03-11T22:43:29.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:29.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822661 2020-03-11T22:43:29.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:29.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:29.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822661 2020-03-11T22:43:29.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:29.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822661 2020-03-11T22:43:29.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822661' 2020-03-11T22:43:29.251 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 154618822661 2020-03-11T22:43:29.251 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:29.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822658 -lt 154618822661 2020-03-11T22:43:29.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:30.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:30.614 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:30.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822661 -lt 154618822661 2020-03-11T22:43:30.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:30.993 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:43:30.993 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:30.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:30.994 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:43:30.994 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:30.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:43:30.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:43:30.995 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:30.996 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 2020-03-11T22:43:31.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672979 -lt 2020-03-11T22:43:31.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672979: unary operator expected 2020-03-11T22:43:31.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:31.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-158913789956 2020-03-11T22:43:31.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:31.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:31.436 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-158913789956 2020-03-11T22:43:31.436 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:31.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789956 2020-03-11T22:43:31.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 158913789956' 2020-03-11T22:43:31.438 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:31.438 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 158913789956 2020-03-11T22:43:31.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789956 -lt 158913789956 2020-03-11T22:43:31.798 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:31.798 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:31.798 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:32.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:32.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:32.279 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:32.279 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:32.279 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:32.279 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:32.280 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:32.280 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:32.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:32.631 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:32.631 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:32.631 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:33.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:33.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:43:33.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:43:33.171 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:43:33.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:33.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:43:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:33.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:33.273 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:33.273 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:33.273 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:33.273 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:33.273 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:33.273 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:33.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:33.624 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:33.625 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:33.625 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:34.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:34.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:34.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:34.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:34.096 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:34.096 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:34.096 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:34.096 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:34.097 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:43:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:34.570 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:43:34.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:34.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:34.770 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:34.770 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:34.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:34.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:34.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:34.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:35.120 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:35.120 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:35.121 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:35.121 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:35.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:35.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:35.594 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:35.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:36.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:36.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:36.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:43:36.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:36.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:43:36.466 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:36.466 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:36.467 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:36.467 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:36.467 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:36.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:36.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:36.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:36.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:36.823 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:36.823 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:36.823 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:37.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:37.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:37.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:37.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:37.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:37.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:37.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:37.293 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:37.293 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:37.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:37.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:37.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:43:37.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:37.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:43:38.563 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:38.563 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:38.564 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:38.564 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:38.565 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:38.565 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:38.565 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:38.565 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:38.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:38.924 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:38.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:38.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:39.399 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:39.399 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:39.399 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:39.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:39.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:39.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:39.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:39.400 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:39.400 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:39.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:39.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:39.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:43:39.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:39.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:43:41.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:41.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:41.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:41.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:41.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:41.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:41.475 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:41.476 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:41.840 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:41.841 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:41.841 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:41.841 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:42.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:42.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:42.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:42.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:42.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:42.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:42.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:42.325 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:42.325 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:42.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:42.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:42.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:43:42.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:42.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:43:46.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:46.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:46.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:46.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:46.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:46.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:46.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:46.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:46.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:46.382 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:46.382 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:46.382 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:46.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:46.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:46.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:46.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:46.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:46.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:46.861 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:46.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:46.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:47.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:47.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:47.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:43:47.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:47.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:43:53.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:53.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:53.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:53.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:53.736 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:53.736 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:53.736 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:53.736 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:54.097 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:54.098 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:54.098 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:54.098 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:54.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:54.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:54.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:54.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:54.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:54.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:54.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:54.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:54.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:55.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:55.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:55.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:43:55.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:55.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:44:02.364 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:02.364 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:02.365 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:02.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:02.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:02.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:02.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:02.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:02.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:02.728 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:02.729 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:02.729 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:03.207 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:03.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:03.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:03.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:03.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:03.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:03.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:03.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:03.209 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:03.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:03.680 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:03.680 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:44:03.680 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:44:04.151 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:44:04.151 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_fingerprint": "04d8008a-eeb6-4e2e-9c93-4054c660cdb1", 2020-03-11T22:44:04.151 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:44:04.151 INFO:tasks.workunit.client.0.smithi016.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:44:04.152 INFO:tasks.workunit.client.0.smithi016.stdout: "timestamp": "2020-03-11T22:44:04.147748+0000", 2020-03-11T22:44:04.152 INFO:tasks.workunit.client.0.smithi016.stdout: "tag": "", 2020-03-11T22:44:04.152 INFO:tasks.workunit.client.0.smithi016.stdout: "health": { 2020-03-11T22:44:04.152 INFO:tasks.workunit.client.0.smithi016.stdout: "status": "HEALTH_WARN", 2020-03-11T22:44:04.152 INFO:tasks.workunit.client.0.smithi016.stdout: "checks": { 2020-03-11T22:44:04.152 INFO:tasks.workunit.client.0.smithi016.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:44:04.153 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:04.153 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:44:04.153 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:44:04.153 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 17 2020-03-11T22:44:04.153 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.153 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [], 2020-03-11T22:44:04.153 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:44:04.154 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.154 INFO:tasks.workunit.client.0.smithi016.stdout: "OSD_DOWN": { 2020-03-11T22:44:04.154 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:04.154 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:44:04.154 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 osds down", 2020-03-11T22:44:04.154 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:44:04.154 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.155 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:44:04.155 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.155 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "osd.1 (root=default,host=smithi016) is down" 2020-03-11T22:44:04.155 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.155 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.155 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:44:04.155 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.156 INFO:tasks.workunit.client.0.smithi016.stdout: "PG_DEGRADED": { 2020-03-11T22:44:04.156 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:04.156 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:44:04.156 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:44:04.156 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:44:04.156 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.156 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:44:04.157 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.157 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:44:04.157 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.157 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.157 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:44:04.157 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.157 INFO:tasks.workunit.client.0.smithi016.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:44:04.158 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:04.158 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:44:04.158 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:44:04.158 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:44:04.158 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.158 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:44:04.158 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.159 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:44:04.159 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.159 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.159 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:44:04.159 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.159 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.159 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:44:04.159 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.160 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.160 INFO:tasks.workunit.client.0.smithi016.stdout: "mutes": [] 2020-03-11T22:44:04.160 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.160 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_first_committed": 1, 2020-03-11T22:44:04.160 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_last_committed": 1, 2020-03-11T22:44:04.160 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap": { 2020-03-11T22:44:04.160 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:44:04.161 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:44:04.161 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:42:11.160366Z", 2020-03-11T22:44:04.161 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:42:11.160366Z", 2020-03-11T22:44:04.161 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release": 15, 2020-03-11T22:44:04.161 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:44:04.162 INFO:tasks.workunit.client.0.smithi016.stdout: "features": { 2020-03-11T22:44:04.162 INFO:tasks.workunit.client.0.smithi016.stdout: "persistent": [ 2020-03-11T22:44:04.162 INFO:tasks.workunit.client.0.smithi016.stdout: "kraken", 2020-03-11T22:44:04.162 INFO:tasks.workunit.client.0.smithi016.stdout: "luminous", 2020-03-11T22:44:04.162 INFO:tasks.workunit.client.0.smithi016.stdout: "mimic", 2020-03-11T22:44:04.162 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap-prune", 2020-03-11T22:44:04.162 INFO:tasks.workunit.client.0.smithi016.stdout: "nautilus", 2020-03-11T22:44:04.163 INFO:tasks.workunit.client.0.smithi016.stdout: "octopus" 2020-03-11T22:44:04.163 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.163 INFO:tasks.workunit.client.0.smithi016.stdout: "optional": [] 2020-03-11T22:44:04.163 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.163 INFO:tasks.workunit.client.0.smithi016.stdout: "mons": [ 2020-03-11T22:44:04.164 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.164 INFO:tasks.workunit.client.0.smithi016.stdout: "rank": 0, 2020-03-11T22:44:04.164 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "a", 2020-03-11T22:44:04.164 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:44:04.164 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.164 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.165 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.165 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:44:04.165 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 0 2020-03-11T22:44:04.165 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.165 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.165 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.166 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:44:04.166 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:44:04.166 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 0, 2020-03-11T22:44:04.166 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 0 2020-03-11T22:44:04.166 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.167 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.167 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.167 INFO:tasks.workunit.client.0.smithi016.stdout: "quorum": [ 2020-03-11T22:44:04.167 INFO:tasks.workunit.client.0.smithi016.stdout: 0 2020-03-11T22:44:04.167 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.167 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap": { 2020-03-11T22:44:04.167 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 40, 2020-03-11T22:44:04.168 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:44:04.168 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:42:11.853243+0000", 2020-03-11T22:44:04.168 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:43:22.859593+0000", 2020-03-11T22:44:04.168 INFO:tasks.workunit.client.0.smithi016.stdout: "last_up_change": "2020-03-11T22:43:21.156968+0000", 2020-03-11T22:44:04.169 INFO:tasks.workunit.client.0.smithi016.stdout: "last_in_change": "2020-03-11T22:43:22.684417+0000", 2020-03-11T22:44:04.169 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:44:04.169 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_num": 5800384, 2020-03-11T22:44:04.169 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_set": [ 2020-03-11T22:44:04.169 INFO:tasks.workunit.client.0.smithi016.stdout: "nodown", 2020-03-11T22:44:04.169 INFO:tasks.workunit.client.0.smithi016.stdout: "noin", 2020-03-11T22:44:04.170 INFO:tasks.workunit.client.0.smithi016.stdout: "noout", 2020-03-11T22:44:04.170 INFO:tasks.workunit.client.0.smithi016.stdout: "pglog_hardlimit", 2020-03-11T22:44:04.170 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snapdirs", 2020-03-11T22:44:04.170 INFO:tasks.workunit.client.0.smithi016.stdout: "recovery_deletes", 2020-03-11T22:44:04.170 INFO:tasks.workunit.client.0.smithi016.stdout: "sortbitwise" 2020-03-11T22:44:04.170 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.170 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_version": 8, 2020-03-11T22:44:04.171 INFO:tasks.workunit.client.0.smithi016.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:44:04.171 INFO:tasks.workunit.client.0.smithi016.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:44:04.171 INFO:tasks.workunit.client.0.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:44:04.171 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_snapshot": "", 2020-03-11T22:44:04.171 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_max": 1, 2020-03-11T22:44:04.171 INFO:tasks.workunit.client.0.smithi016.stdout: "max_osd": 3, 2020-03-11T22:44:04.171 INFO:tasks.workunit.client.0.smithi016.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:44:04.171 INFO:tasks.workunit.client.0.smithi016.stdout: "min_compat_client": "jewel", 2020-03-11T22:44:04.172 INFO:tasks.workunit.client.0.smithi016.stdout: "require_osd_release": "octopus", 2020-03-11T22:44:04.172 INFO:tasks.workunit.client.0.smithi016.stdout: "pools": [ 2020-03-11T22:44:04.172 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.172 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:44:04.172 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_name": "test", 2020-03-11T22:44:04.172 INFO:tasks.workunit.client.0.smithi016.stdout: "create_time": "2020-03-11T22:42:34.877702+0000", 2020-03-11T22:44:04.172 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": 16389, 2020-03-11T22:44:04.173 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:44:04.173 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 3, 2020-03-11T22:44:04.173 INFO:tasks.workunit.client.0.smithi016.stdout: "size": 3, 2020-03-11T22:44:04.173 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 2, 2020-03-11T22:44:04.173 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_rule": 1, 2020-03-11T22:44:04.173 INFO:tasks.workunit.client.0.smithi016.stdout: "object_hash": 2, 2020-03-11T22:44:04.173 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:44:04.174 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num": 1, 2020-03-11T22:44:04.174 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num": 1, 2020-03-11T22:44:04.174 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num_target": 1, 2020-03-11T22:44:04.174 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_target": 1, 2020-03-11T22:44:04.174 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_pending": 1, 2020-03-11T22:44:04.174 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pg_merge_meta": { 2020-03-11T22:44:04.174 INFO:tasks.workunit.client.0.smithi016.stdout: "source_pgid": "0.0", 2020-03-11T22:44:04.175 INFO:tasks.workunit.client.0.smithi016.stdout: "ready_epoch": 0, 2020-03-11T22:44:04.175 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_started": 0, 2020-03-11T22:44:04.175 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_clean": 0, 2020-03-11T22:44:04.175 INFO:tasks.workunit.client.0.smithi016.stdout: "source_version": "0'0", 2020-03-11T22:44:04.175 INFO:tasks.workunit.client.0.smithi016.stdout: "target_version": "0'0" 2020-03-11T22:44:04.175 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.175 INFO:tasks.workunit.client.0.smithi016.stdout: "last_change": "30", 2020-03-11T22:44:04.176 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend": "0", 2020-03-11T22:44:04.176 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:44:04.176 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:44:04.176 INFO:tasks.workunit.client.0.smithi016.stdout: "auid": 0, 2020-03-11T22:44:04.176 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_mode": "pool", 2020-03-11T22:44:04.176 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_seq": 1, 2020-03-11T22:44:04.176 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_epoch": 30, 2020-03-11T22:44:04.177 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_snaps": [ 2020-03-11T22:44:04.177 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.177 INFO:tasks.workunit.client.0.smithi016.stdout: "snapid": 1, 2020-03-11T22:44:04.177 INFO:tasks.workunit.client.0.smithi016.stdout: "stamp": "2020-03-11T22:43:03.661174+0000", 2020-03-11T22:44:04.177 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "snap1" 2020-03-11T22:44:04.177 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.177 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.178 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps": "[]", 2020-03-11T22:44:04.178 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_bytes": 0, 2020-03-11T22:44:04.178 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_objects": 0, 2020-03-11T22:44:04.178 INFO:tasks.workunit.client.0.smithi016.stdout: "tiers": [], 2020-03-11T22:44:04.178 INFO:tasks.workunit.client.0.smithi016.stdout: "tier_of": -1, 2020-03-11T22:44:04.178 INFO:tasks.workunit.client.0.smithi016.stdout: "read_tier": -1, 2020-03-11T22:44:04.178 INFO:tasks.workunit.client.0.smithi016.stdout: "write_tier": -1, 2020-03-11T22:44:04.179 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_mode": "none", 2020-03-11T22:44:04.179 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_bytes": 0, 2020-03-11T22:44:04.179 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_objects": 0, 2020-03-11T22:44:04.179 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:44:04.179 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:44:04.179 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:44:04.179 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_flush_age": 0, 2020-03-11T22:44:04.180 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_evict_age": 0, 2020-03-11T22:44:04.180 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:44:04.180 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_params": { 2020-03-11T22:44:04.180 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "none" 2020-03-11T22:44:04.180 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.180 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_period": 0, 2020-03-11T22:44:04.180 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_count": 0, 2020-03-11T22:44:04.181 INFO:tasks.workunit.client.0.smithi016.stdout: "use_gmt_hitset": true, 2020-03-11T22:44:04.181 INFO:tasks.workunit.client.0.smithi016.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:44:04.181 INFO:tasks.workunit.client.0.smithi016.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:44:04.181 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:44:04.181 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:44:04.181 INFO:tasks.workunit.client.0.smithi016.stdout: "grade_table": [], 2020-03-11T22:44:04.182 INFO:tasks.workunit.client.0.smithi016.stdout: "stripe_width": 8192, 2020-03-11T22:44:04.182 INFO:tasks.workunit.client.0.smithi016.stdout: "expected_num_objects": 0, 2020-03-11T22:44:04.182 INFO:tasks.workunit.client.0.smithi016.stdout: "fast_read": false, 2020-03-11T22:44:04.182 INFO:tasks.workunit.client.0.smithi016.stdout: "options": {}, 2020-03-11T22:44:04.182 INFO:tasks.workunit.client.0.smithi016.stdout: "application_metadata": {} 2020-03-11T22:44:04.182 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.182 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.182 INFO:tasks.workunit.client.0.smithi016.stdout: "osds": [ 2020-03-11T22:44:04.183 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.183 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:44:04.183 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "6c73bd18-95db-4897-9cb7-6ae241013376", 2020-03-11T22:44:04.183 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:44:04.183 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:44:04.183 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:44:04.183 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:44:04.184 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 5, 2020-03-11T22:44:04.184 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 29, 2020-03-11T22:44:04.184 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 36, 2020-03-11T22:44:04.184 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 39, 2020-03-11T22:44:04.184 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 31, 2020-03-11T22:44:04.184 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:44:04.184 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:44:04.185 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.185 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.185 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.185 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:44:04.185 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48703 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48703 2020-03-11T22:44:04.187 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.187 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.187 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.187 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:44:04.187 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.188 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.188 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.188 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:44:04.188 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48703 2020-03-11T22:44:04.188 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.188 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.188 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.189 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:44:04.189 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48703 2020-03-11T22:44:04.189 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.189 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.189 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.189 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:04.190 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.190 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.190 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.190 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:44:04.190 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48703 2020-03-11T22:44:04.190 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.190 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.190 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.191 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:44:04.191 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48703 2020-03-11T22:44:04.191 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.191 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.191 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.191 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:04.191 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.192 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.192 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.192 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:44:04.192 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48703 2020-03-11T22:44:04.192 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.192 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.192 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.193 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:44:04.193 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48703 2020-03-11T22:44:04.193 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.193 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.193 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.193 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6803/48703", 2020-03-11T22:44:04.193 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6805/48703", 2020-03-11T22:44:04.194 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6809/48703", 2020-03-11T22:44:04.194 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6807/48703", 2020-03-11T22:44:04.194 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:44:04.194 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:44:04.194 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:44:04.194 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.194 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "6e1b8749-4a8a-4a6e-8118-60aab81e6f12", 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 0, 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 0, 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 0, 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 10, 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 25, 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 32, 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:44:04.197 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.197 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.197 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.197 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:04.197 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:04.198 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.198 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.198 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.198 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:04.199 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:04.199 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.199 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.199 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.199 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:44:04.199 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.200 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.200 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.200 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:04.200 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:04.201 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.201 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.201 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.201 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:04.201 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:04.202 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.202 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.202 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.202 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:04.202 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.203 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.203 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.203 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:04.203 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:04.203 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.204 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.204 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.204 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:04.204 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:04.204 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.204 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.205 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.205 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:04.205 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.205 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.205 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.205 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:04.205 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:04.206 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.206 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.206 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.206 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:04.206 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:04.206 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.206 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.207 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.207 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/43944", 2020-03-11T22:44:04.207 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/43944", 2020-03-11T22:44:04.207 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43944", 2020-03-11T22:44:04.207 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43944", 2020-03-11T22:44:04.207 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:44:04.208 INFO:tasks.workunit.client.0.smithi016.stdout: "exists" 2020-03-11T22:44:04.208 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.208 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.208 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.208 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:44:04.208 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "6d353a96-21d2-42eb-bd15-df39bc2f3fbb", 2020-03-11T22:44:04.208 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:44:04.209 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:44:04.209 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:44:04.209 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:44:04.209 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 15, 2020-03-11T22:44:04.209 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 29, 2020-03-11T22:44:04.209 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 37, 2020-03-11T22:44:04.209 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 0, 2020-03-11T22:44:04.210 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 33, 2020-03-11T22:44:04.210 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:44:04.210 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:44:04.210 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.210 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.210 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.210 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:04.211 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48897 2020-03-11T22:44:04.211 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.211 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.211 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.211 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:04.211 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48897 2020-03-11T22:44:04.211 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.212 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.212 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.212 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:44:04.212 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.212 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.212 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.212 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:04.212 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48897 2020-03-11T22:44:04.213 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.213 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.213 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.213 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:04.213 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48897 2020-03-11T22:44:04.213 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.213 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.214 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.214 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:04.214 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.214 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.214 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.214 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:04.214 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48897 2020-03-11T22:44:04.215 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.215 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.215 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.215 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:04.215 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48897 2020-03-11T22:44:04.215 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.215 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.216 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.216 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:04.216 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:04.216 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.216 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:04.216 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:04.216 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48897 2020-03-11T22:44:04.216 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.217 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.217 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:04.217 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:04.217 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 48897 2020-03-11T22:44:04.217 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.218 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.218 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.218 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/48897", 2020-03-11T22:44:04.218 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/48897", 2020-03-11T22:44:04.218 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48897", 2020-03-11T22:44:04.218 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48897", 2020-03-11T22:44:04.219 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:44:04.219 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:44:04.219 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:44:04.219 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.219 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.219 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.219 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_xinfo": [ 2020-03-11T22:44:04.220 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.220 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:44:04.220 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:43:14.601295+0000", 2020-03-11T22:44:04.220 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:44:04.220 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:44:04.220 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:44:04.220 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:44:04.220 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:18.953587+0000", 2020-03-11T22:44:04.221 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:44:04.221 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.221 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.221 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:44:04.221 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:43:15.179045+0000", 2020-03-11T22:44:04.221 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:44:04.221 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:44:04.222 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:44:04.222 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:44:04.222 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:25.366507+0000", 2020-03-11T22:44:04.222 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:44:04.222 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.222 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.222 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:44:04.223 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:43:15.762882+0000", 2020-03-11T22:44:04.223 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:44:04.223 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:44:04.223 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:44:04.223 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:44:04.223 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:31.773812+0000", 2020-03-11T22:44:04.223 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:44:04.224 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.224 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.224 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap": [], 2020-03-11T22:44:04.224 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap_items": [], 2020-03-11T22:44:04.224 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_temp": [], 2020-03-11T22:44:04.224 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_temp": [], 2020-03-11T22:44:04.224 INFO:tasks.workunit.client.0.smithi016.stdout: "blacklist": {}, 2020-03-11T22:44:04.224 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profiles": { 2020-03-11T22:44:04.225 INFO:tasks.workunit.client.0.smithi016.stdout: "default": { 2020-03-11T22:44:04.225 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:44:04.225 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "2", 2020-03-11T22:44:04.225 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:44:04.225 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van" 2020-03-11T22:44:04.225 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.225 INFO:tasks.workunit.client.0.smithi016.stdout: "myprofile": { 2020-03-11T22:44:04.226 INFO:tasks.workunit.client.0.smithi016.stdout: "crush-device-class": "", 2020-03-11T22:44:04.226 INFO:tasks.workunit.client.0.smithi016.stdout: "crush-failure-domain": "osd", 2020-03-11T22:44:04.226 INFO:tasks.workunit.client.0.smithi016.stdout: "crush-root": "default", 2020-03-11T22:44:04.226 INFO:tasks.workunit.client.0.smithi016.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:44:04.226 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:44:04.226 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "1", 2020-03-11T22:44:04.226 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:44:04.227 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van", 2020-03-11T22:44:04.227 INFO:tasks.workunit.client.0.smithi016.stdout: "w": "8" 2020-03-11T22:44:04.227 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.227 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.227 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps_queue": [], 2020-03-11T22:44:04.227 INFO:tasks.workunit.client.0.smithi016.stdout: "new_removed_snaps": [], 2020-03-11T22:44:04.227 INFO:tasks.workunit.client.0.smithi016.stdout: "new_purged_snaps": [], 2020-03-11T22:44:04.228 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_node_flags": {}, 2020-03-11T22:44:04.228 INFO:tasks.workunit.client.0.smithi016.stdout: "device_class_flags": {} 2020-03-11T22:44:04.228 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.228 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_metadata": [ 2020-03-11T22:44:04.228 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.228 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:44:04.228 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:44:04.229 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6804/48703,v1:127.0.0.1:6805/48703]", 2020-03-11T22:44:04.229 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:44:04.229 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:44:04.229 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:04.229 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:04.229 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:04.229 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:04.230 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:04.230 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:04.230 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:04.230 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:44:04.230 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:04.230 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:04.230 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:04.230 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:04.231 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:44:04.231 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:04.231 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:04.231 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:04.231 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:44:04.231 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:44:04.231 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:04.232 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:44:04.232 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:44:04.232 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:04.232 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:44:04.232 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6802/48703,v1:127.0.0.1:6803/48703]", 2020-03-11T22:44:04.232 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:44:04.233 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/48703,v1:127.0.0.1:6809/48703]", 2020-03-11T22:44:04.233 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/48703,v1:127.0.0.1:6807/48703]", 2020-03-11T22:44:04.233 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:44:04.233 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:44:04.233 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:04.233 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:04.234 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:04.234 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:44:04.234 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:04.234 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:04.234 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:44:04.235 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:44:04.235 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:04.235 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:44:04.235 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.235 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.235 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:44:04.235 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:44:04.235 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/43944,v1:127.0.0.1:6813/43944]", 2020-03-11T22:44:04.236 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:44:04.236 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:44:04.236 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:04.236 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:04.236 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:04.236 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:04.236 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:04.237 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:04.237 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:04.237 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:44:04.237 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:04.237 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:04.237 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:04.237 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:04.238 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:44:04.238 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:04.238 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:04.238 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:04.238 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:44:04.238 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:44:04.238 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:04.239 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:44:04.239 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:44:04.239 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:04.239 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:44:04.239 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/43944,v1:127.0.0.1:6811/43944]", 2020-03-11T22:44:04.239 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:44:04.239 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43944,v1:127.0.0.1:6817/43944]", 2020-03-11T22:44:04.239 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43944,v1:127.0.0.1:6815/43944]", 2020-03-11T22:44:04.240 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:44:04.240 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:44:04.240 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:04.240 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:04.240 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:04.240 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:44:04.240 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:04.241 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:04.241 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:44:04.241 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:44:04.241 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:04.241 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:44:04.241 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.241 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.241 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:44:04.242 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:44:04.242 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/48897,v1:127.0.0.1:6813/48897]", 2020-03-11T22:44:04.242 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:44:04.242 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:44:04.242 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:04.242 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:04.242 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:04.243 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:04.243 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:04.243 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:04.243 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:04.243 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:44:04.243 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:04.243 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:04.244 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:04.244 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:04.244 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:44:04.244 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:04.244 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:04.244 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:04.244 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:44:04.245 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:44:04.245 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:04.245 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:44:04.246 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:44:04.246 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:04.246 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:44:04.246 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/48897,v1:127.0.0.1:6811/48897]", 2020-03-11T22:44:04.246 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:44:04.246 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48897,v1:127.0.0.1:6817/48897]", 2020-03-11T22:44:04.247 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48897,v1:127.0.0.1:6815/48897]", 2020-03-11T22:44:04.247 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:44:04.247 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:44:04.247 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:04.247 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:04.247 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:04.247 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:44:04.248 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:04.248 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:04.248 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:44:04.248 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:44:04.248 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:04.248 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:44:04.248 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.249 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.249 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_first_committed": 1, 2020-03-11T22:44:04.249 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_last_committed": 40, 2020-03-11T22:44:04.249 INFO:tasks.workunit.client.0.smithi016.stdout: "crushmap": { 2020-03-11T22:44:04.249 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": [ 2020-03-11T22:44:04.249 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.249 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:44:04.250 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.0", 2020-03-11T22:44:04.250 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:44:04.250 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.250 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.250 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:44:04.250 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.1", 2020-03-11T22:44:04.250 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:44:04.251 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.251 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.251 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:44:04.251 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.2", 2020-03-11T22:44:04.251 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:44:04.251 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.251 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.252 INFO:tasks.workunit.client.0.smithi016.stdout: "types": [ 2020-03-11T22:44:04.252 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.252 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 0, 2020-03-11T22:44:04.252 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd" 2020-03-11T22:44:04.252 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.252 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.252 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:44:04.252 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "host" 2020-03-11T22:44:04.253 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.253 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.253 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 2, 2020-03-11T22:44:04.253 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "chassis" 2020-03-11T22:44:04.253 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.253 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 3, 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "rack" 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 4, 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "row" 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 5, 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pdu" 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 6, 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pod" 2020-03-11T22:44:04.256 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.256 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.256 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 7, 2020-03-11T22:44:04.256 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "room" 2020-03-11T22:44:04.256 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.256 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.256 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 8, 2020-03-11T22:44:04.257 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "datacenter" 2020-03-11T22:44:04.257 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.257 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.257 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 9, 2020-03-11T22:44:04.257 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "zone" 2020-03-11T22:44:04.257 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.257 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.257 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 10, 2020-03-11T22:44:04.258 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "region" 2020-03-11T22:44:04.258 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.258 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.258 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:44:04.258 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "root" 2020-03-11T22:44:04.258 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.258 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.259 INFO:tasks.workunit.client.0.smithi016.stdout: "buckets": [ 2020-03-11T22:44:04.259 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.259 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -1, 2020-03-11T22:44:04.259 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default", 2020-03-11T22:44:04.259 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:44:04.259 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:44:04.259 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:04.259 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:44:04.260 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:44:04.260 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:44:04.260 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.260 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:44:04.260 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:04.260 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:44:04.260 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.261 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.261 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.261 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.261 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -2, 2020-03-11T22:44:04.261 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default~hdd", 2020-03-11T22:44:04.261 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:44:04.262 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:44:04.262 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:04.262 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:44:04.262 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:44:04.262 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:44:04.262 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.262 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:44:04.263 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:04.263 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:44:04.263 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.263 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.263 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.263 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.264 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:44:04.264 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016", 2020-03-11T22:44:04.264 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:44:04.264 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:44:04.264 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:04.264 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:44:04.264 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:44:04.265 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:44:04.265 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.265 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:44:04.265 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:04.265 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:44:04.266 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.266 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.266 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:44:04.266 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:04.266 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:44:04.266 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.266 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.267 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:44:04.267 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:04.267 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:44:04.267 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.267 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.267 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.267 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.268 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:44:04.268 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016~hdd", 2020-03-11T22:44:04.268 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:44:04.268 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:44:04.268 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:04.268 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:44:04.268 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:44:04.269 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:44:04.269 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.269 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:44:04.269 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:04.269 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:44:04.269 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.269 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.270 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:44:04.270 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:04.270 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:44:04.270 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.270 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.270 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:44:04.271 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:04.271 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:44:04.271 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.271 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.271 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.271 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.271 INFO:tasks.workunit.client.0.smithi016.stdout: "rules": [ 2020-03-11T22:44:04.272 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.272 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 0, 2020-03-11T22:44:04.272 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "replicated_rule", 2020-03-11T22:44:04.272 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 0, 2020-03-11T22:44:04.272 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:44:04.272 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 1, 2020-03-11T22:44:04.272 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 10, 2020-03-11T22:44:04.272 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.273 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_firstn", 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.274 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 1, 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "test", 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 1, 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 3, 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 3, 2020-03-11T22:44:04.275 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 3, 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 5 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.276 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "set_choose_tries", 2020-03-11T22:44:04.277 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 100 2020-03-11T22:44:04.277 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.277 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.277 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:44:04.277 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:44:04.277 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:44:04.277 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.278 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.278 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_indep", 2020-03-11T22:44:04.278 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:44:04.278 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:44:04.278 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.278 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.278 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:44:04.279 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.279 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:04.279 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.279 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.279 INFO:tasks.workunit.client.0.smithi016.stdout: "tunables": { 2020-03-11T22:44:04.279 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_tries": 0, 2020-03-11T22:44:04.279 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:44:04.279 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_total_tries": 50, 2020-03-11T22:44:04.280 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:44:04.280 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:44:04.280 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_stable": 1, 2020-03-11T22:44:04.280 INFO:tasks.workunit.client.0.smithi016.stdout: "straw_calc_version": 1, 2020-03-11T22:44:04.280 INFO:tasks.workunit.client.0.smithi016.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:44:04.280 INFO:tasks.workunit.client.0.smithi016.stdout: "profile": "jewel", 2020-03-11T22:44:04.280 INFO:tasks.workunit.client.0.smithi016.stdout: "optimal_tunables": 1, 2020-03-11T22:44:04.281 INFO:tasks.workunit.client.0.smithi016.stdout: "legacy_tunables": 0, 2020-03-11T22:44:04.281 INFO:tasks.workunit.client.0.smithi016.stdout: "minimum_required_version": "jewel", 2020-03-11T22:44:04.281 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables": 1, 2020-03-11T22:44:04.281 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables2": 1, 2020-03-11T22:44:04.281 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v2_rules": 1, 2020-03-11T22:44:04.281 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables3": 1, 2020-03-11T22:44:04.281 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v3_rules": 0, 2020-03-11T22:44:04.282 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v4_buckets": 1, 2020-03-11T22:44:04.282 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables5": 1, 2020-03-11T22:44:04.282 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v5_rules": 0 2020-03-11T22:44:04.282 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.282 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_args": {} 2020-03-11T22:44:04.282 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.282 INFO:tasks.workunit.client.0.smithi016.stdout: "fsmap": { 2020-03-11T22:44:04.283 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:44:04.283 INFO:tasks.workunit.client.0.smithi016.stdout: "default_fscid": -1, 2020-03-11T22:44:04.283 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": { 2020-03-11T22:44:04.283 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": {}, 2020-03-11T22:44:04.283 INFO:tasks.workunit.client.0.smithi016.stdout: "ro_compat": {}, 2020-03-11T22:44:04.283 INFO:tasks.workunit.client.0.smithi016.stdout: "incompat": { 2020-03-11T22:44:04.283 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_1": "base v0.20", 2020-03-11T22:44:04.283 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:44:04.284 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:44:04.284 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:44:04.284 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:44:04.284 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:44:04.284 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_8": "no anchor table", 2020-03-11T22:44:04.284 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_9": "file layout v2", 2020-03-11T22:44:04.284 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:44:04.285 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.285 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.285 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_flags": { 2020-03-11T22:44:04.285 INFO:tasks.workunit.client.0.smithi016.stdout: "enable_multiple": false, 2020-03-11T22:44:04.285 INFO:tasks.workunit.client.0.smithi016.stdout: "ever_enabled_multiple": false 2020-03-11T22:44:04.285 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.285 INFO:tasks.workunit.client.0.smithi016.stdout: "standbys": [], 2020-03-11T22:44:04.286 INFO:tasks.workunit.client.0.smithi016.stdout: "filesystems": [] 2020-03-11T22:44:04.286 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.286 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:44:04.286 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:44:04.286 INFO:tasks.workunit.client.0.smithi016.stdout: "auth": { 2020-03-11T22:44:04.286 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:44:04.286 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 5, 2020-03-11T22:44:04.287 INFO:tasks.workunit.client.0.smithi016.stdout: "num_secrets": 3 2020-03-11T22:44:04.287 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.287 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:44:04.287 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_active": 1, 2020-03-11T22:44:04.287 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_unknown": 0, 2020-03-11T22:44:04.287 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osd": 3, 2020-03-11T22:44:04.287 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_sum": { 2020-03-11T22:44:04.287 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:44:04.288 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48430, 2020-03-11T22:44:04.288 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 101, 2020-03-11T22:44:04.288 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 1, 2020-03-11T22:44:04.288 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 303, 2020-03-11T22:44:04.288 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:04.288 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:44:04.288 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:04.289 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:04.289 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:04.289 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:04.289 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:44:04.289 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:44:04.289 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:44:04.289 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 101, 2020-03-11T22:44:04.290 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 101, 2020-03-11T22:44:04.290 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:04.290 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:04.290 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:04.290 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:04.290 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:04.290 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:04.291 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:44:04.291 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:04.291 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:04.291 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:44:04.291 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:44:04.291 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:44:04.291 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:44:04.292 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:44:04.292 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:04.292 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:04.292 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:04.292 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:04.292 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:04.292 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:04.293 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:04.293 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:04.293 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:04.293 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:44:04.293 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:44:04.293 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.293 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:44:04.294 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:44:04.294 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:44:04.294 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:44:04.294 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 0, 2020-03-11T22:44:04.294 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 0, 2020-03-11T22:44:04.294 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:44:04.294 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:04.295 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:44:04.295 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:44:04.295 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:44:04.295 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.295 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 18, 2020-03-11T22:44:04.295 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:04.296 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 3, 2020-03-11T22:44:04.296 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 3, 2020-03-11T22:44:04.296 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 0 2020-03-11T22:44:04.296 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.296 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum": { 2020-03-11T22:44:04.296 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:44:04.297 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:44:04.297 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:44:04.297 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:44:04.297 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:04.297 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:04.297 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:44:04.298 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3179904, 2020-03-11T22:44:04.298 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 33856, 2020-03-11T22:44:04.298 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 11, 2020-03-11T22:44:04.298 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145716, 2020-03-11T22:44:04.298 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311392896, 2020-03-11T22:44:04.298 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:44:04.298 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:44:04.299 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318866325504, 2020-03-11T22:44:04.299 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:04.299 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 34668544, 2020-03-11T22:44:04.299 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 1523781, 2020-03-11T22:44:04.300 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:44:04.300 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:04.300 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:44:04.300 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 11560, 2020-03-11T22:44:04.300 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221213912 2020-03-11T22:44:04.300 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.301 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:44:04.301 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:04.301 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:04.301 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:04.301 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:44:04.301 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:44:04.303 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.303 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:44:04.303 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:44:04.303 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.303 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum_by_class": { 2020-03-11T22:44:04.303 INFO:tasks.workunit.client.0.smithi016.stdout: "hdd": { 2020-03-11T22:44:04.303 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:44:04.303 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:44:04.304 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:44:04.304 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:44:04.304 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:04.304 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:04.304 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:44:04.304 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3179904, 2020-03-11T22:44:04.304 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 33856, 2020-03-11T22:44:04.305 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 11, 2020-03-11T22:44:04.305 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145716, 2020-03-11T22:44:04.305 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311392896, 2020-03-11T22:44:04.305 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:44:04.305 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:44:04.305 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318866325504, 2020-03-11T22:44:04.306 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:04.306 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 34668544, 2020-03-11T22:44:04.306 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 1523781, 2020-03-11T22:44:04.306 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:44:04.306 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:04.307 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:44:04.307 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 11560, 2020-03-11T22:44:04.307 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221213912 2020-03-11T22:44:04.307 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.307 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:44:04.307 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:04.308 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:04.308 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:04.308 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:44:04.308 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:44:04.308 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:44:04.308 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.308 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:44:04.309 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:04.309 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:04.309 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:04.309 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:44:04.309 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.309 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:44:04.309 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:44:04.310 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.310 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.310 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_stats": [ 2020-03-11T22:44:04.310 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.310 INFO:tasks.workunit.client.0.smithi016.stdout: "poolid": 1, 2020-03-11T22:44:04.310 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:44:04.310 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:44:04.311 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48430, 2020-03-11T22:44:04.311 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 101, 2020-03-11T22:44:04.311 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 1, 2020-03-11T22:44:04.311 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 303, 2020-03-11T22:44:04.311 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:04.311 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:44:04.311 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:04.311 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:04.312 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:04.312 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:04.312 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:44:04.312 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:44:04.312 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:44:04.312 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 101, 2020-03-11T22:44:04.312 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 101, 2020-03-11T22:44:04.313 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:04.313 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:04.313 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:04.313 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:04.313 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:04.314 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:04.314 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:44:04.314 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:04.315 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:04.315 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:44:04.315 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:44:04.315 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:44:04.316 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:44:04.316 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:44:04.316 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:04.316 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:04.316 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:04.317 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:04.317 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:04.317 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:04.317 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:04.317 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:04.318 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:04.318 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:44:04.318 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:44:04.318 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.318 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:44:04.319 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:44:04.319 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:44:04.319 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:44:04.319 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 19857408, 2020-03-11T22:44:04.320 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 1241088, 2020-03-11T22:44:04.320 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:44:04.320 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:04.320 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:44:04.321 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:44:04.321 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:44:04.321 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.321 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 18, 2020-03-11T22:44:04.321 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:04.322 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 3, 2020-03-11T22:44:04.322 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 3, 2020-03-11T22:44:04.322 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 3 2020-03-11T22:44:04.322 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.322 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.323 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_stats": [ 2020-03-11T22:44:04.323 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.323 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:44:04.323 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 154618822667 2020-03-11T22:44:04.324 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.324 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.324 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:44:04.324 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 42949672979 2020-03-11T22:44:04.324 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.325 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.325 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:44:04.325 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 158913789963 2020-03-11T22:44:04.325 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.325 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.325 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_state": [ 2020-03-11T22:44:04.325 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.326 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "active+undersized+degraded", 2020-03-11T22:44:04.326 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 1 2020-03-11T22:44:04.326 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.326 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.328 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_osd": [ 2020-03-11T22:44:04.329 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.329 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:44:04.329 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:44:04.329 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:44:04.329 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:04.329 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.330 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.330 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2147483647, 2020-03-11T22:44:04.330 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:44:04.330 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:44:04.330 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:04.330 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.330 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.331 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:44:04.331 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:44:04.331 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 0, 2020-03-11T22:44:04.331 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:04.331 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.331 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.331 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:44:04.332 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 1, 2020-03-11T22:44:04.332 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:44:04.332 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:04.332 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.332 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.332 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": [ 2020-03-11T22:44:04.332 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:04.333 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:44:04.333 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": {} 2020-03-11T22:44:04.333 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.333 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:04.333 INFO:tasks.workunit.client.0.smithi016.stdout: "servicemap": { 2020-03-11T22:44:04.334 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:44:04.334 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:42:11.853755+0000", 2020-03-11T22:44:04.334 INFO:tasks.workunit.client.0.smithi016.stdout: "services": {} 2020-03-11T22:44:04.334 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:04.334 INFO:tasks.workunit.client.0.smithi016.stdout: "paxos": { 2020-03-11T22:44:04.334 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:44:04.335 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 333, 2020-03-11T22:44:04.335 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pn": 0, 2020-03-11T22:44:04.335 INFO:tasks.workunit.client.0.smithi016.stdout: "accepted_pn": 0 2020-03-11T22:44:04.335 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:04.335 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:44:04.337 INFO:tasks.workunit.client.0.smithi016.stderr:report 1499272476 2020-03-11T22:44:04.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:44:04.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:04.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-03-11T22:44:04.338 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:04.338 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:04.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:04.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:04.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:04.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:04.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-03-11T22:44:04.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:44:04.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:44:04.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-03-11T22:44:04.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:44:04.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-03-11T22:44:04.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-03-11T22:44:04.827 INFO:tasks.workunit.client.0.smithi016.stderr:PG '1.0' not found 2020-03-11T22:44:05.349 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:44:05.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:44:05.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:05.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:05.350 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:44:05.351 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:05.351 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:44:05.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:05.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:05.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:05.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:05.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:44:05.354 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:44:05.354 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:05.354 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:05.354 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:05.354 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:05.355 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:05.355 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:05.355 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:05.355 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:05.356 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:44:05.356 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:44:05.356 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:44:05.356 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:05.357 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:05.357 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:05.357 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:05.357 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:05.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:05.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:05.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:05.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:44:05.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:44:05.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:05.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:44:05.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:44:05.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:05.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:05.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:05.360 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:44:05.378 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:05.374+0000 7f59e6f67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:05.379 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:05.376+0000 7f59e6f67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:05.381 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:05.378+0000 7f59e6f67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:05.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:44:05.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:05.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:05.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:05.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:05.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:05.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:05.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:05.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:05.730 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:44:05.902 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:05.898+0000 7f59e6f67ec0 -1 Falling back to public interface 2020-03-11T22:44:06.104 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 36 up_thru 39 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/48703,v1:127.0.0.1:6803/48703] [v2:127.0.0.1:6804/48703,v1:127.0.0.1:6805/48703] exists,up 6c73bd18-95db-4897-9cb7-6ae241013376 2020-03-11T22:44:06.104 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:06.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:06.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:06.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:06.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-03-11T22:44:06.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:06.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:06.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:06.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:06.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:06.209 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:06.209 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-03-11T22:44:06.209 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:44:06.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:44:06.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-03-11T22:44:06.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:44:06.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-03-11T22:44:06.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-03-11T22:44:06.822 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:06.819+0000 7f59e6f67ec0 -1 osd.0 40 log_to_monitors {default=true} 2020-03-11T22:44:06.993 INFO:tasks.workunit.client.0.smithi016.stderr:PG '1.0' not found 2020-03-11T22:44:07.513 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:44:07.513 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:44:07.513 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:07.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:07.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:44:07.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:07.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:44:07.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:07.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:07.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:07.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:07.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:44:07.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:44:07.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:07.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:07.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:07.515 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:07.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:07.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:07.516 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:07.516 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:07.516 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:44:07.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:44:07.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:44:07.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:07.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:07.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:07.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:07.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:07.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:07.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:07.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:07.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:44:07.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:44:07.518 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:44:07.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:07.519 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:44:07.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:44:07.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:07.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:07.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:07.545 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:07.542+0000 7f9540bb1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:07.547 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:07.544+0000 7f9540bb1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:07.549 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:07.546+0000 7f9540bb1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:07.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:44:07.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:07.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:07.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:07.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:07.889 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:07.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:07.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:07.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:07.890 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:44:08.070 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:08.066+0000 7f9540bb1ec0 -1 Falling back to public interface 2020-03-11T22:44:08.331 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/48897,v1:127.0.0.1:6811/48897] [v2:127.0.0.1:6812/48897,v1:127.0.0.1:6813/48897] exists,up 6d353a96-21d2-42eb-bd15-df39bc2f3fbb 2020-03-11T22:44:08.331 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:08.331 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:08.331 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:08.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-03-11T22:44:08.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:44:08.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:44:08.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:44:08.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:44:08.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:44:08.332 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-03-11T22:44:09.021 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:09.018+0000 7f9540bb1ec0 -1 osd.2 40 log_to_monitors {default=true} 2020-03-11T22:44:09.069 INFO:tasks.workunit.client.0.smithi016.stderr:PG '1.0' not found 2020-03-11T22:44:09.590 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:44:09.590 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:09.590 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:09.590 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:09.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:09.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:09.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:09.591 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:09.592 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:09.592 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:09.592 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:09.592 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:09.592 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:09.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:09.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:09.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:09.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:09.661 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:10.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:10.029 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:44:10.029 INFO:tasks.workunit.client.0.smithi016.stderr:2' 2020-03-11T22:44:10.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:10.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:10.030 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:10.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=180388626435 2020-03-11T22:44:10.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435' 2020-03-11T22:44:10.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:10.166 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:10.262 INFO:tasks.workunit.client.0.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:44:10.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:44:10.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435 1-' 2020-03-11T22:44:10.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:10.272 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:10.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=193273528323 2020-03-11T22:44:10.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435 1- 2-193273528323' 2020-03-11T22:44:10.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:10.407 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-180388626435 2020-03-11T22:44:10.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:10.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:10.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-180388626435 2020-03-11T22:44:10.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:10.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=180388626435 2020-03-11T22:44:10.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 180388626435' 2020-03-11T22:44:10.410 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:10.410 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 180388626435 2020-03-11T22:44:10.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 180388626435 2020-03-11T22:44:10.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:11.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:11.772 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:12.148 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 180388626435 2020-03-11T22:44:12.148 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:13.149 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:44:13.150 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:13.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 180388626436 -lt 180388626435 2020-03-11T22:44:13.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:13.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:44:13.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:13.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:13.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:44:13.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:13.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:44:13.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:44:13.529 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:13.529 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 2020-03-11T22:44:13.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672979 -lt 2020-03-11T22:44:13.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672979: unary operator expected 2020-03-11T22:44:13.891 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:13.891 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-193273528323 2020-03-11T22:44:13.891 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:13.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:13.892 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-193273528323 2020-03-11T22:44:13.892 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:13.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=193273528323 2020-03-11T22:44:13.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 193273528323' 2020-03-11T22:44:13.894 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 193273528323 2020-03-11T22:44:13.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:14.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 193273528324 -lt 193273528323 2020-03-11T22:44:14.292 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:14.293 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:14.293 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:14.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:14.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:14.778 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:14.778 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:14.778 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:14.779 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:14.779 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:14.779 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:15.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:15.129 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:15.129 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:15.130 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:15.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:15.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:44:15.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:44:15.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:44:15.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:15.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:44:15.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:15.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:15.708 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:15.708 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:15.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:15.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:15.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:15.709 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:16.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:16.061 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:16.062 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:16.062 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:16.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:16.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:16.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:16.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:16.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:16.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:16.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:16.532 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:16.532 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:17.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:17.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:17.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:44:17.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:17.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:44:17.207 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:17.207 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:17.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:17.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:17.209 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:17.209 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:17.209 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:17.209 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:17.564 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:17.566 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:17.566 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:17.567 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:18.033 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:18.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:18.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:18.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:18.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:18.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:18.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:18.035 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:18.035 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:18.504 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:18.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:18.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:44:18.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:18.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:44:18.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:18.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:18.906 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:18.907 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:18.907 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:18.907 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:18.907 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:18.907 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:19.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:19.267 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:19.268 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:19.268 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:19.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:19.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:19.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:19.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:19.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:19.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:19.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:19.728 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:19.728 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:20.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:20.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:20.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:44:20.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:20.222 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:44:21.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:21.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:21.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:21.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:21.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:21.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:21.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:21.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:21.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:21.386 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:21.386 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:21.387 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:21.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:21.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:21.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:21.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:21.859 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:21.859 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:21.859 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:21.859 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:21.859 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:22.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:22.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:22.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:44:22.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:22.330 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:44:23.930 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:23.930 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:23.931 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:23.931 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:23.931 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:23.932 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:23.932 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:23.932 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:24.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:24.298 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:24.298 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:24.299 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:24.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:24.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:24.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:24.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:24.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:24.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:24.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:24.772 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:24.772 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:25.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:25.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:25.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:44:25.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:25.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:44:28.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:28.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:28.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:28.455 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:28.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:28.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:28.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:28.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:28.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:28.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:28.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:28.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:29.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:29.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:29.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:29.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:29.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:29.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:29.288 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:29.288 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:29.288 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:29.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:29.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:29.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:44:29.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:29.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:44:36.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:36.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:36.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:36.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:36.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:36.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:36.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:36.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:36.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:36.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:36.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:36.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:36.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:36.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:36.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:37.000 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:37.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:37.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:37.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:44:37.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:37.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:44:44.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:44.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:44.775 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:44.775 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:44.775 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:44.775 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:44.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:44.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:45.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:45.139 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:45.140 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:45.140 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:45.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:45.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:45.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:45.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:45.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:45.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:45.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:45.616 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:45.616 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:46.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:46.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:46.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:44:46.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:44:46.632 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:44:46.632 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_fingerprint": "04d8008a-eeb6-4e2e-9c93-4054c660cdb1", 2020-03-11T22:44:46.633 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:44:46.633 INFO:tasks.workunit.client.0.smithi016.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:44:46.633 INFO:tasks.workunit.client.0.smithi016.stdout: "timestamp": "2020-03-11T22:44:46.629104+0000", 2020-03-11T22:44:46.633 INFO:tasks.workunit.client.0.smithi016.stdout: "tag": "", 2020-03-11T22:44:46.633 INFO:tasks.workunit.client.0.smithi016.stdout: "health": { 2020-03-11T22:44:46.634 INFO:tasks.workunit.client.0.smithi016.stdout: "status": "HEALTH_WARN", 2020-03-11T22:44:46.634 INFO:tasks.workunit.client.0.smithi016.stdout: "checks": { 2020-03-11T22:44:46.634 INFO:tasks.workunit.client.0.smithi016.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:44:46.634 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:46.634 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:44:46.634 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:44:46.634 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 17 2020-03-11T22:44:46.635 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.635 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [], 2020-03-11T22:44:46.635 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:44:46.635 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.635 INFO:tasks.workunit.client.0.smithi016.stdout: "OSD_DOWN": { 2020-03-11T22:44:46.635 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:46.636 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:44:46.636 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 osds down", 2020-03-11T22:44:46.636 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:44:46.636 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.636 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:44:46.636 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.636 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "osd.1 (root=default,host=smithi016) is down" 2020-03-11T22:44:46.637 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.637 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.639 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:44:46.639 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.639 INFO:tasks.workunit.client.0.smithi016.stdout: "PG_DEGRADED": { 2020-03-11T22:44:46.639 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:46.639 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:44:46.639 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:44:46.640 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:44:46.640 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.640 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:44:46.640 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.640 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:44:46.640 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.641 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.641 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:44:46.641 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.641 INFO:tasks.workunit.client.0.smithi016.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:44:46.641 INFO:tasks.workunit.client.0.smithi016.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:46.642 INFO:tasks.workunit.client.0.smithi016.stdout: "summary": { 2020-03-11T22:44:46.642 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:44:46.642 INFO:tasks.workunit.client.0.smithi016.stdout: "count": 1 2020-03-11T22:44:46.642 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.642 INFO:tasks.workunit.client.0.smithi016.stdout: "detail": [ 2020-03-11T22:44:46.642 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.645 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:44:46.645 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.645 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.646 INFO:tasks.workunit.client.0.smithi016.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:44:46.646 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.646 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.646 INFO:tasks.workunit.client.0.smithi016.stdout: "muted": false 2020-03-11T22:44:46.646 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.646 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.646 INFO:tasks.workunit.client.0.smithi016.stdout: "mutes": [] 2020-03-11T22:44:46.647 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.647 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_first_committed": 1, 2020-03-11T22:44:46.647 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap_last_committed": 1, 2020-03-11T22:44:46.647 INFO:tasks.workunit.client.0.smithi016.stdout: "monmap": { 2020-03-11T22:44:46.647 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:44:46.647 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:44:46.648 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:42:11.160366Z", 2020-03-11T22:44:46.648 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:42:11.160366Z", 2020-03-11T22:44:46.648 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release": 15, 2020-03-11T22:44:46.648 INFO:tasks.workunit.client.0.smithi016.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:44:46.648 INFO:tasks.workunit.client.0.smithi016.stdout: "features": { 2020-03-11T22:44:46.648 INFO:tasks.workunit.client.0.smithi016.stdout: "persistent": [ 2020-03-11T22:44:46.649 INFO:tasks.workunit.client.0.smithi016.stdout: "kraken", 2020-03-11T22:44:46.649 INFO:tasks.workunit.client.0.smithi016.stdout: "luminous", 2020-03-11T22:44:46.649 INFO:tasks.workunit.client.0.smithi016.stdout: "mimic", 2020-03-11T22:44:46.649 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap-prune", 2020-03-11T22:44:46.649 INFO:tasks.workunit.client.0.smithi016.stdout: "nautilus", 2020-03-11T22:44:46.649 INFO:tasks.workunit.client.0.smithi016.stdout: "octopus" 2020-03-11T22:44:46.649 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.650 INFO:tasks.workunit.client.0.smithi016.stdout: "optional": [] 2020-03-11T22:44:46.650 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.650 INFO:tasks.workunit.client.0.smithi016.stdout: "mons": [ 2020-03-11T22:44:46.650 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.650 INFO:tasks.workunit.client.0.smithi016.stdout: "rank": 0, 2020-03-11T22:44:46.650 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "a", 2020-03-11T22:44:46.650 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:44:46.650 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.651 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.651 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.651 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:44:46.651 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 0 2020-03-11T22:44:46.651 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.652 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.652 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.653 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:44:46.653 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:44:46.653 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 0, 2020-03-11T22:44:46.653 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 0 2020-03-11T22:44:46.653 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.653 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.653 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.654 INFO:tasks.workunit.client.0.smithi016.stdout: "quorum": [ 2020-03-11T22:44:46.654 INFO:tasks.workunit.client.0.smithi016.stdout: 0 2020-03-11T22:44:46.654 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.654 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap": { 2020-03-11T22:44:46.655 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 46, 2020-03-11T22:44:46.655 INFO:tasks.workunit.client.0.smithi016.stdout: "fsid": "c7572dbb-081c-44c5-977c-66ff16469e9f", 2020-03-11T22:44:46.655 INFO:tasks.workunit.client.0.smithi016.stdout: "created": "2020-03-11T22:42:11.853243+0000", 2020-03-11T22:44:46.655 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:44:09.544110+0000", 2020-03-11T22:44:46.655 INFO:tasks.workunit.client.0.smithi016.stdout: "last_up_change": "2020-03-11T22:44:09.309189+0000", 2020-03-11T22:44:46.655 INFO:tasks.workunit.client.0.smithi016.stdout: "last_in_change": "2020-03-11T22:43:22.684417+0000", 2020-03-11T22:44:46.655 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:44:46.656 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_num": 5800384, 2020-03-11T22:44:46.656 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_set": [ 2020-03-11T22:44:46.656 INFO:tasks.workunit.client.0.smithi016.stdout: "nodown", 2020-03-11T22:44:46.656 INFO:tasks.workunit.client.0.smithi016.stdout: "noin", 2020-03-11T22:44:46.656 INFO:tasks.workunit.client.0.smithi016.stdout: "noout", 2020-03-11T22:44:46.656 INFO:tasks.workunit.client.0.smithi016.stdout: "pglog_hardlimit", 2020-03-11T22:44:46.656 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snapdirs", 2020-03-11T22:44:46.657 INFO:tasks.workunit.client.0.smithi016.stdout: "recovery_deletes", 2020-03-11T22:44:46.657 INFO:tasks.workunit.client.0.smithi016.stdout: "sortbitwise" 2020-03-11T22:44:46.657 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.657 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_version": 8, 2020-03-11T22:44:46.657 INFO:tasks.workunit.client.0.smithi016.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:44:46.657 INFO:tasks.workunit.client.0.smithi016.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:44:46.658 INFO:tasks.workunit.client.0.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:44:46.658 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_snapshot": "", 2020-03-11T22:44:46.658 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_max": 1, 2020-03-11T22:44:46.658 INFO:tasks.workunit.client.0.smithi016.stdout: "max_osd": 3, 2020-03-11T22:44:46.658 INFO:tasks.workunit.client.0.smithi016.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:44:46.658 INFO:tasks.workunit.client.0.smithi016.stdout: "min_compat_client": "jewel", 2020-03-11T22:44:46.659 INFO:tasks.workunit.client.0.smithi016.stdout: "require_osd_release": "octopus", 2020-03-11T22:44:46.659 INFO:tasks.workunit.client.0.smithi016.stdout: "pools": [ 2020-03-11T22:44:46.659 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.659 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:44:46.659 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_name": "test", 2020-03-11T22:44:46.659 INFO:tasks.workunit.client.0.smithi016.stdout: "create_time": "2020-03-11T22:42:34.877702+0000", 2020-03-11T22:44:46.659 INFO:tasks.workunit.client.0.smithi016.stdout: "flags": 16389, 2020-03-11T22:44:46.659 INFO:tasks.workunit.client.0.smithi016.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:44:46.660 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 3, 2020-03-11T22:44:46.660 INFO:tasks.workunit.client.0.smithi016.stdout: "size": 3, 2020-03-11T22:44:46.660 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 2, 2020-03-11T22:44:46.660 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_rule": 1, 2020-03-11T22:44:46.660 INFO:tasks.workunit.client.0.smithi016.stdout: "object_hash": 2, 2020-03-11T22:44:46.660 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:44:46.661 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num": 1, 2020-03-11T22:44:46.661 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num": 1, 2020-03-11T22:44:46.661 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_placement_num_target": 1, 2020-03-11T22:44:46.661 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_target": 1, 2020-03-11T22:44:46.661 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_num_pending": 1, 2020-03-11T22:44:46.661 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pg_merge_meta": { 2020-03-11T22:44:46.661 INFO:tasks.workunit.client.0.smithi016.stdout: "source_pgid": "0.0", 2020-03-11T22:44:46.662 INFO:tasks.workunit.client.0.smithi016.stdout: "ready_epoch": 0, 2020-03-11T22:44:46.662 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_started": 0, 2020-03-11T22:44:46.662 INFO:tasks.workunit.client.0.smithi016.stdout: "last_epoch_clean": 0, 2020-03-11T22:44:46.663 INFO:tasks.workunit.client.0.smithi016.stdout: "source_version": "0'0", 2020-03-11T22:44:46.663 INFO:tasks.workunit.client.0.smithi016.stdout: "target_version": "0'0" 2020-03-11T22:44:46.663 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.664 INFO:tasks.workunit.client.0.smithi016.stdout: "last_change": "30", 2020-03-11T22:44:46.664 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend": "0", 2020-03-11T22:44:46.664 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:44:46.664 INFO:tasks.workunit.client.0.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:44:46.664 INFO:tasks.workunit.client.0.smithi016.stdout: "auid": 0, 2020-03-11T22:44:46.664 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_mode": "pool", 2020-03-11T22:44:46.665 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_seq": 1, 2020-03-11T22:44:46.665 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_epoch": 30, 2020-03-11T22:44:46.665 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_snaps": [ 2020-03-11T22:44:46.665 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.665 INFO:tasks.workunit.client.0.smithi016.stdout: "snapid": 1, 2020-03-11T22:44:46.666 INFO:tasks.workunit.client.0.smithi016.stdout: "stamp": "2020-03-11T22:43:03.661174+0000", 2020-03-11T22:44:46.666 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "snap1" 2020-03-11T22:44:46.666 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.666 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.666 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps": "[]", 2020-03-11T22:44:46.666 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_bytes": 0, 2020-03-11T22:44:46.666 INFO:tasks.workunit.client.0.smithi016.stdout: "quota_max_objects": 0, 2020-03-11T22:44:46.667 INFO:tasks.workunit.client.0.smithi016.stdout: "tiers": [], 2020-03-11T22:44:46.667 INFO:tasks.workunit.client.0.smithi016.stdout: "tier_of": -1, 2020-03-11T22:44:46.667 INFO:tasks.workunit.client.0.smithi016.stdout: "read_tier": -1, 2020-03-11T22:44:46.667 INFO:tasks.workunit.client.0.smithi016.stdout: "write_tier": -1, 2020-03-11T22:44:46.667 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_mode": "none", 2020-03-11T22:44:46.667 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_bytes": 0, 2020-03-11T22:44:46.668 INFO:tasks.workunit.client.0.smithi016.stdout: "target_max_objects": 0, 2020-03-11T22:44:46.668 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:44:46.668 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:44:46.668 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:44:46.668 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_flush_age": 0, 2020-03-11T22:44:46.668 INFO:tasks.workunit.client.0.smithi016.stdout: "cache_min_evict_age": 0, 2020-03-11T22:44:46.669 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:44:46.669 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_params": { 2020-03-11T22:44:46.669 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "none" 2020-03-11T22:44:46.669 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.669 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_period": 0, 2020-03-11T22:44:46.669 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_count": 0, 2020-03-11T22:44:46.669 INFO:tasks.workunit.client.0.smithi016.stdout: "use_gmt_hitset": true, 2020-03-11T22:44:46.670 INFO:tasks.workunit.client.0.smithi016.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:44:46.670 INFO:tasks.workunit.client.0.smithi016.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:44:46.670 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:44:46.670 INFO:tasks.workunit.client.0.smithi016.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:44:46.670 INFO:tasks.workunit.client.0.smithi016.stdout: "grade_table": [], 2020-03-11T22:44:46.670 INFO:tasks.workunit.client.0.smithi016.stdout: "stripe_width": 8192, 2020-03-11T22:44:46.671 INFO:tasks.workunit.client.0.smithi016.stdout: "expected_num_objects": 0, 2020-03-11T22:44:46.671 INFO:tasks.workunit.client.0.smithi016.stdout: "fast_read": false, 2020-03-11T22:44:46.671 INFO:tasks.workunit.client.0.smithi016.stdout: "options": {}, 2020-03-11T22:44:46.671 INFO:tasks.workunit.client.0.smithi016.stdout: "application_metadata": {} 2020-03-11T22:44:46.671 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.671 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.672 INFO:tasks.workunit.client.0.smithi016.stdout: "osds": [ 2020-03-11T22:44:46.672 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.672 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:44:46.672 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "6c73bd18-95db-4897-9cb7-6ae241013376", 2020-03-11T22:44:46.672 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:44:46.672 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:44:46.673 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:44:46.673 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:44:46.673 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 36, 2020-03-11T22:44:46.673 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 40, 2020-03-11T22:44:46.673 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 42, 2020-03-11T22:44:46.673 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 45, 2020-03-11T22:44:46.675 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 41, 2020-03-11T22:44:46.675 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:44:46.676 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:44:46.676 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.676 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.677 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.677 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:44:46.677 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50346 2020-03-11T22:44:46.677 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.677 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.677 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.678 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:44:46.678 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50346 2020-03-11T22:44:46.678 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.678 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.678 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.678 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:44:46.679 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.679 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.679 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.679 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:44:46.679 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50346 2020-03-11T22:44:46.679 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.679 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.680 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.680 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:44:46.680 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50346 2020-03-11T22:44:46.680 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.680 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.680 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.680 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:46.681 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.681 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.681 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.681 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:44:46.681 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50346 2020-03-11T22:44:46.681 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.682 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.682 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.682 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:44:46.683 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50346 2020-03-11T22:44:46.683 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.683 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.683 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.683 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:46.683 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.683 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.684 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.684 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:44:46.684 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50346 2020-03-11T22:44:46.684 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.684 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.684 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.684 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:44:46.685 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50346 2020-03-11T22:44:46.685 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.686 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.686 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.686 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6803/50346", 2020-03-11T22:44:46.686 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6805/50346", 2020-03-11T22:44:46.686 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6809/50346", 2020-03-11T22:44:46.686 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6807/50346", 2020-03-11T22:44:46.687 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:44:46.687 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:44:46.687 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:44:46.687 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.687 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.687 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.687 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:44:46.688 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "6e1b8749-4a8a-4a6e-8118-60aab81e6f12", 2020-03-11T22:44:46.688 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 0, 2020-03-11T22:44:46.688 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:44:46.688 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:44:46.688 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:44:46.688 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 0, 2020-03-11T22:44:46.688 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 0, 2020-03-11T22:44:46.688 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 10, 2020-03-11T22:44:46.689 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 25, 2020-03-11T22:44:46.689 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 32, 2020-03-11T22:44:46.689 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:44:46.689 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:44:46.689 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.689 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.690 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.690 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:46.690 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:46.690 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.690 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.691 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.691 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:46.691 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:46.691 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.691 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.691 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.694 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:46.694 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.694 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.694 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.694 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:46.694 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:46.694 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.695 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.695 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.695 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:46.695 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:46.695 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.695 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.695 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.696 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:46.696 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.696 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.696 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.696 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:46.696 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:46.697 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.697 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.697 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.697 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:46.697 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 43944 2020-03-11T22:44:46.697 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.697 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.698 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.698 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/43944", 2020-03-11T22:44:46.698 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/43944", 2020-03-11T22:44:46.698 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43944", 2020-03-11T22:44:46.698 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43944", 2020-03-11T22:44:46.698 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:44:46.698 INFO:tasks.workunit.client.0.smithi016.stdout: "exists" 2020-03-11T22:44:46.699 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.699 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.699 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.699 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:44:46.699 INFO:tasks.workunit.client.0.smithi016.stdout: "uuid": "6d353a96-21d2-42eb-bd15-df39bc2f3fbb", 2020-03-11T22:44:46.699 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 1, 2020-03-11T22:44:46.699 INFO:tasks.workunit.client.0.smithi016.stdout: "in": 1, 2020-03-11T22:44:46.700 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 1, 2020-03-11T22:44:46.700 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_affinity": 1, 2020-03-11T22:44:46.700 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_begin": 37, 2020-03-11T22:44:46.700 INFO:tasks.workunit.client.0.smithi016.stdout: "last_clean_end": 40, 2020-03-11T22:44:46.700 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 45, 2020-03-11T22:44:46.701 INFO:tasks.workunit.client.0.smithi016.stdout: "up_thru": 0, 2020-03-11T22:44:46.701 INFO:tasks.workunit.client.0.smithi016.stdout: "down_at": 44, 2020-03-11T22:44:46.701 INFO:tasks.workunit.client.0.smithi016.stdout: "lost_at": 0, 2020-03-11T22:44:46.701 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addrs": { 2020-03-11T22:44:46.701 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.701 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.702 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.702 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:46.702 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50578 2020-03-11T22:44:46.702 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.702 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.702 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.702 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:46.703 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50578 2020-03-11T22:44:46.703 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.703 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.703 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.703 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addrs": { 2020-03-11T22:44:46.703 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.704 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.704 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.704 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:46.704 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50578 2020-03-11T22:44:46.704 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.704 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.704 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.705 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:46.705 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50578 2020-03-11T22:44:46.705 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.705 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.705 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.705 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:46.705 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.706 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.706 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.706 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:46.706 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50578 2020-03-11T22:44:46.706 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.706 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50578 2020-03-11T22:44:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:46.708 INFO:tasks.workunit.client.0.smithi016.stdout: "addrvec": [ 2020-03-11T22:44:46.708 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.709 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v2", 2020-03-11T22:44:46.709 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:46.709 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50578 2020-03-11T22:44:46.709 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.709 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.709 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "v1", 2020-03-11T22:44:46.710 INFO:tasks.workunit.client.0.smithi016.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:46.710 INFO:tasks.workunit.client.0.smithi016.stdout: "nonce": 50578 2020-03-11T22:44:46.710 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.710 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.710 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.710 INFO:tasks.workunit.client.0.smithi016.stdout: "public_addr": "127.0.0.1:6811/50578", 2020-03-11T22:44:46.710 INFO:tasks.workunit.client.0.smithi016.stdout: "cluster_addr": "127.0.0.1:6813/50578", 2020-03-11T22:44:46.710 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_back_addr": "127.0.0.1:6817/50578", 2020-03-11T22:44:46.711 INFO:tasks.workunit.client.0.smithi016.stdout: "heartbeat_front_addr": "127.0.0.1:6815/50578", 2020-03-11T22:44:46.711 INFO:tasks.workunit.client.0.smithi016.stdout: "state": [ 2020-03-11T22:44:46.711 INFO:tasks.workunit.client.0.smithi016.stdout: "exists", 2020-03-11T22:44:46.711 INFO:tasks.workunit.client.0.smithi016.stdout: "up" 2020-03-11T22:44:46.711 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.711 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.711 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.712 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_xinfo": [ 2020-03-11T22:44:46.712 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.712 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:44:46.712 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:44:06.951684+0000", 2020-03-11T22:44:46.713 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:44:46.713 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:44:46.713 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:44:46.713 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:44:46.713 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:18.953587+0000", 2020-03-11T22:44:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:44:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:44:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:43:15.179045+0000", 2020-03-11T22:44:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:44:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:44:46.714 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:44:46.715 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:44:46.715 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:25.366507+0000", 2020-03-11T22:44:46.715 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:44:46.715 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.715 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.715 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:44:46.715 INFO:tasks.workunit.client.0.smithi016.stdout: "down_stamp": "2020-03-11T22:44:09.100526+0000", 2020-03-11T22:44:46.716 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_probability": 0, 2020-03-11T22:44:46.716 INFO:tasks.workunit.client.0.smithi016.stdout: "laggy_interval": 0, 2020-03-11T22:44:46.716 INFO:tasks.workunit.client.0.smithi016.stdout: "features": 4540138292836696063, 2020-03-11T22:44:46.716 INFO:tasks.workunit.client.0.smithi016.stdout: "old_weight": 0, 2020-03-11T22:44:46.716 INFO:tasks.workunit.client.0.smithi016.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:31.773812+0000", 2020-03-11T22:44:46.716 INFO:tasks.workunit.client.0.smithi016.stdout: "dead_epoch": 0 2020-03-11T22:44:46.716 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.717 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.717 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap": [], 2020-03-11T22:44:46.717 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_upmap_items": [], 2020-03-11T22:44:46.717 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_temp": [], 2020-03-11T22:44:46.717 INFO:tasks.workunit.client.0.smithi016.stdout: "primary_temp": [], 2020-03-11T22:44:46.717 INFO:tasks.workunit.client.0.smithi016.stdout: "blacklist": {}, 2020-03-11T22:44:46.717 INFO:tasks.workunit.client.0.smithi016.stdout: "erasure_code_profiles": { 2020-03-11T22:44:46.718 INFO:tasks.workunit.client.0.smithi016.stdout: "default": { 2020-03-11T22:44:46.718 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:44:46.718 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "2", 2020-03-11T22:44:46.718 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:44:46.718 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van" 2020-03-11T22:44:46.718 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.718 INFO:tasks.workunit.client.0.smithi016.stdout: "myprofile": { 2020-03-11T22:44:46.719 INFO:tasks.workunit.client.0.smithi016.stdout: "crush-device-class": "", 2020-03-11T22:44:46.719 INFO:tasks.workunit.client.0.smithi016.stdout: "crush-failure-domain": "osd", 2020-03-11T22:44:46.719 INFO:tasks.workunit.client.0.smithi016.stdout: "crush-root": "default", 2020-03-11T22:44:46.719 INFO:tasks.workunit.client.0.smithi016.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:44:46.719 INFO:tasks.workunit.client.0.smithi016.stdout: "k": "2", 2020-03-11T22:44:46.719 INFO:tasks.workunit.client.0.smithi016.stdout: "m": "1", 2020-03-11T22:44:46.720 INFO:tasks.workunit.client.0.smithi016.stdout: "plugin": "jerasure", 2020-03-11T22:44:46.720 INFO:tasks.workunit.client.0.smithi016.stdout: "technique": "reed_sol_van", 2020-03-11T22:44:46.720 INFO:tasks.workunit.client.0.smithi016.stdout: "w": "8" 2020-03-11T22:44:46.720 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.720 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.720 INFO:tasks.workunit.client.0.smithi016.stdout: "removed_snaps_queue": [], 2020-03-11T22:44:46.720 INFO:tasks.workunit.client.0.smithi016.stdout: "new_removed_snaps": [], 2020-03-11T22:44:46.721 INFO:tasks.workunit.client.0.smithi016.stdout: "new_purged_snaps": [], 2020-03-11T22:44:46.721 INFO:tasks.workunit.client.0.smithi016.stdout: "crush_node_flags": {}, 2020-03-11T22:44:46.721 INFO:tasks.workunit.client.0.smithi016.stdout: "device_class_flags": {} 2020-03-11T22:44:46.721 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.721 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_metadata": [ 2020-03-11T22:44:46.721 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.721 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:44:46.722 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:44:46.722 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6804/50346,v1:127.0.0.1:6805/50346]", 2020-03-11T22:44:46.722 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:44:46.722 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:44:46.722 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:46.722 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:46.722 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:46.723 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:46.723 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:46.723 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:46.723 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:46.723 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:44:46.723 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:46.723 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:46.723 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:46.724 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:46.724 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:44:46.724 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:46.724 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:46.724 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:46.724 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:44:46.724 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:44:46.725 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:46.725 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:44:46.725 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:44:46.725 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:46.725 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:44:46.725 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6802/50346,v1:127.0.0.1:6803/50346]", 2020-03-11T22:44:46.725 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:44:46.726 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/50346,v1:127.0.0.1:6809/50346]", 2020-03-11T22:44:46.726 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/50346,v1:127.0.0.1:6807/50346]", 2020-03-11T22:44:46.726 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:44:46.726 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:44:46.726 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:46.726 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:46.726 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:46.727 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:44:46.727 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:46.727 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:46.727 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:44:46.727 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:44:46.727 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:46.727 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:44:46.728 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.728 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.728 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:44:46.728 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:44:46.728 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/43944,v1:127.0.0.1:6813/43944]", 2020-03-11T22:44:46.728 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:44:46.728 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:44:46.729 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:46.729 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:46.729 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:46.729 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:46.729 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:46.729 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:46.729 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:46.730 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:44:46.730 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:46.730 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:46.730 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:46.730 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:46.730 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:44:46.730 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:46.731 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:46.731 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:46.731 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:44:46.731 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:44:46.731 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:46.731 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:44:46.732 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:44:46.732 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:46.732 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:44:46.732 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/43944,v1:127.0.0.1:6811/43944]", 2020-03-11T22:44:46.732 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:44:46.732 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43944,v1:127.0.0.1:6817/43944]", 2020-03-11T22:44:46.732 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43944,v1:127.0.0.1:6815/43944]", 2020-03-11T22:44:46.733 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:44:46.733 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:44:46.733 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:46.733 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:46.733 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:46.733 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:44:46.733 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:46.733 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:46.734 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:44:46.734 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:44:46.734 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:46.734 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:44:46.734 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.734 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.735 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:44:46.735 INFO:tasks.workunit.client.0.smithi016.stdout: "arch": "x86_64", 2020-03-11T22:44:46.735 INFO:tasks.workunit.client.0.smithi016.stdout: "back_addr": "[v2:127.0.0.1:6812/50578,v1:127.0.0.1:6813/50578]", 2020-03-11T22:44:46.735 INFO:tasks.workunit.client.0.smithi016.stdout: "back_iface": "", 2020-03-11T22:44:46.735 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs": "1", 2020-03-11T22:44:46.735 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:46.736 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:46.736 INFO:tasks.workunit.client.0.smithi016.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:46.736 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:46.736 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:46.736 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:46.736 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:46.736 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:44:46.737 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:46.737 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:46.737 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:46.737 INFO:tasks.workunit.client.0.smithi016.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:46.737 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_release": "octopus", 2020-03-11T22:44:46.737 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:46.737 INFO:tasks.workunit.client.0.smithi016.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:46.738 INFO:tasks.workunit.client.0.smithi016.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:44:46.738 INFO:tasks.workunit.client.0.smithi016.stdout: "default_device_class": "hdd", 2020-03-11T22:44:46.738 INFO:tasks.workunit.client.0.smithi016.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HKZ5", 2020-03-11T22:44:46.738 INFO:tasks.workunit.client.0.smithi016.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:46.738 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": "sda", 2020-03-11T22:44:46.738 INFO:tasks.workunit.client.0.smithi016.stdout: "distro": "centos", 2020-03-11T22:44:46.738 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:46.739 INFO:tasks.workunit.client.0.smithi016.stdout: "distro_version": "8", 2020-03-11T22:44:46.739 INFO:tasks.workunit.client.0.smithi016.stdout: "front_addr": "[v2:127.0.0.1:6810/50578,v1:127.0.0.1:6811/50578]", 2020-03-11T22:44:46.739 INFO:tasks.workunit.client.0.smithi016.stdout: "front_iface": "", 2020-03-11T22:44:46.739 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/50578,v1:127.0.0.1:6817/50578]", 2020-03-11T22:44:46.739 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/50578,v1:127.0.0.1:6815/50578]", 2020-03-11T22:44:46.739 INFO:tasks.workunit.client.0.smithi016.stdout: "hostname": "smithi016", 2020-03-11T22:44:46.739 INFO:tasks.workunit.client.0.smithi016.stdout: "journal_rotational": "1", 2020-03-11T22:44:46.740 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:46.740 INFO:tasks.workunit.client.0.smithi016.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:46.740 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:46.740 INFO:tasks.workunit.client.0.smithi016.stdout: "mem_total_kb": "32662096", 2020-03-11T22:44:46.741 INFO:tasks.workunit.client.0.smithi016.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:46.741 INFO:tasks.workunit.client.0.smithi016.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:46.741 INFO:tasks.workunit.client.0.smithi016.stdout: "os": "Linux", 2020-03-11T22:44:46.741 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:44:46.741 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:46.741 INFO:tasks.workunit.client.0.smithi016.stdout: "rotational": "1" 2020-03-11T22:44:46.741 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.742 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.742 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_first_committed": 1, 2020-03-11T22:44:46.742 INFO:tasks.workunit.client.0.smithi016.stdout: "osdmap_last_committed": 46, 2020-03-11T22:44:46.742 INFO:tasks.workunit.client.0.smithi016.stdout: "crushmap": { 2020-03-11T22:44:46.742 INFO:tasks.workunit.client.0.smithi016.stdout: "devices": [ 2020-03-11T22:44:46.742 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.742 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:44:46.743 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.0", 2020-03-11T22:44:46.743 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:44:46.743 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.743 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.743 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:44:46.743 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.1", 2020-03-11T22:44:46.743 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:44:46.744 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.744 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.744 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:44:46.744 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd.2", 2020-03-11T22:44:46.744 INFO:tasks.workunit.client.0.smithi016.stdout: "class": "hdd" 2020-03-11T22:44:46.744 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.744 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.745 INFO:tasks.workunit.client.0.smithi016.stdout: "types": [ 2020-03-11T22:44:46.745 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.745 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 0, 2020-03-11T22:44:46.745 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "osd" 2020-03-11T22:44:46.745 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.745 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.745 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:44:46.746 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "host" 2020-03-11T22:44:46.746 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.746 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.746 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 2, 2020-03-11T22:44:46.746 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "chassis" 2020-03-11T22:44:46.746 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.746 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.747 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 3, 2020-03-11T22:44:46.747 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "rack" 2020-03-11T22:44:46.747 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.747 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.747 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 4, 2020-03-11T22:44:46.747 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "row" 2020-03-11T22:44:46.747 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.748 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.748 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 5, 2020-03-11T22:44:46.748 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pdu" 2020-03-11T22:44:46.748 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.748 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.748 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 6, 2020-03-11T22:44:46.748 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "pod" 2020-03-11T22:44:46.748 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.749 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.749 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 7, 2020-03-11T22:44:46.749 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "room" 2020-03-11T22:44:46.749 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.749 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.749 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 8, 2020-03-11T22:44:46.750 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "datacenter" 2020-03-11T22:44:46.750 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.750 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.750 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 9, 2020-03-11T22:44:46.750 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "zone" 2020-03-11T22:44:46.750 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.750 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.751 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 10, 2020-03-11T22:44:46.751 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "region" 2020-03-11T22:44:46.751 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.751 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.751 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:44:46.751 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "root" 2020-03-11T22:44:46.751 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.751 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.752 INFO:tasks.workunit.client.0.smithi016.stdout: "buckets": [ 2020-03-11T22:44:46.752 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.752 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -1, 2020-03-11T22:44:46.752 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default", 2020-03-11T22:44:46.752 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:44:46.752 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:44:46.752 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:46.753 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:44:46.753 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:44:46.753 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:44:46.753 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.753 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:44:46.753 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:46.753 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:44:46.754 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.754 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.754 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.754 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.754 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -2, 2020-03-11T22:44:46.754 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "default~hdd", 2020-03-11T22:44:46.754 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 11, 2020-03-11T22:44:46.755 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "root", 2020-03-11T22:44:46.755 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:46.755 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:44:46.755 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:44:46.756 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:44:46.756 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.756 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:44:46.756 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:46.756 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:44:46.757 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.757 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.757 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.757 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.757 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -3, 2020-03-11T22:44:46.757 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016", 2020-03-11T22:44:46.757 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:44:46.757 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:44:46.758 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:46.758 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:44:46.758 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:44:46.758 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:44:46.758 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.758 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:44:46.758 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:46.759 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:44:46.759 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.759 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.759 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:44:46.760 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:46.760 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:44:46.760 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.761 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.761 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:44:46.762 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:46.762 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:44:46.762 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.762 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.763 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.763 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.763 INFO:tasks.workunit.client.0.smithi016.stdout: "id": -4, 2020-03-11T22:44:46.763 INFO:tasks.workunit.client.0.smithi016.stdout: "name": "smithi016~hdd", 2020-03-11T22:44:46.763 INFO:tasks.workunit.client.0.smithi016.stdout: "type_id": 1, 2020-03-11T22:44:46.764 INFO:tasks.workunit.client.0.smithi016.stdout: "type_name": "host", 2020-03-11T22:44:46.764 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 19206, 2020-03-11T22:44:46.764 INFO:tasks.workunit.client.0.smithi016.stdout: "alg": "straw2", 2020-03-11T22:44:46.764 INFO:tasks.workunit.client.0.smithi016.stdout: "hash": "rjenkins1", 2020-03-11T22:44:46.764 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:44:46.765 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.765 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 0, 2020-03-11T22:44:46.765 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:46.765 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 0 2020-03-11T22:44:46.766 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.766 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.766 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 1, 2020-03-11T22:44:46.766 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:46.766 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 1 2020-03-11T22:44:46.767 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.767 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.767 INFO:tasks.workunit.client.0.smithi016.stdout: "id": 2, 2020-03-11T22:44:46.767 INFO:tasks.workunit.client.0.smithi016.stdout: "weight": 6402, 2020-03-11T22:44:46.768 INFO:tasks.workunit.client.0.smithi016.stdout: "pos": 2 2020-03-11T22:44:46.768 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.768 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.768 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.768 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.769 INFO:tasks.workunit.client.0.smithi016.stdout: "rules": [ 2020-03-11T22:44:46.769 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.769 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 0, 2020-03-11T22:44:46.769 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "replicated_rule", 2020-03-11T22:44:46.769 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 0, 2020-03-11T22:44:46.769 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 1, 2020-03-11T22:44:46.770 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 1, 2020-03-11T22:44:46.770 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 10, 2020-03-11T22:44:46.770 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:44:46.770 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.771 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:44:46.771 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:44:46.771 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:44:46.771 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.771 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.772 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_firstn", 2020-03-11T22:44:46.772 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:44:46.772 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:44:46.772 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.773 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.773 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:44:46.773 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.774 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.774 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.774 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.774 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_id": 1, 2020-03-11T22:44:46.774 INFO:tasks.workunit.client.0.smithi016.stdout: "rule_name": "test", 2020-03-11T22:44:46.774 INFO:tasks.workunit.client.0.smithi016.stdout: "ruleset": 1, 2020-03-11T22:44:46.775 INFO:tasks.workunit.client.0.smithi016.stdout: "type": 3, 2020-03-11T22:44:46.775 INFO:tasks.workunit.client.0.smithi016.stdout: "min_size": 3, 2020-03-11T22:44:46.775 INFO:tasks.workunit.client.0.smithi016.stdout: "max_size": 3, 2020-03-11T22:44:46.775 INFO:tasks.workunit.client.0.smithi016.stdout: "steps": [ 2020-03-11T22:44:46.776 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.776 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:44:46.776 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 5 2020-03-11T22:44:46.776 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.777 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.777 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "set_choose_tries", 2020-03-11T22:44:46.777 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 100 2020-03-11T22:44:46.777 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.777 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.778 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "take", 2020-03-11T22:44:46.778 INFO:tasks.workunit.client.0.smithi016.stdout: "item": -1, 2020-03-11T22:44:46.778 INFO:tasks.workunit.client.0.smithi016.stdout: "item_name": "default" 2020-03-11T22:44:46.779 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.779 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.779 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "choose_indep", 2020-03-11T22:44:46.779 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 0, 2020-03-11T22:44:46.779 INFO:tasks.workunit.client.0.smithi016.stdout: "type": "osd" 2020-03-11T22:44:46.780 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.780 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.780 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "emit" 2020-03-11T22:44:46.780 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.781 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:44:46.781 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.781 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.781 INFO:tasks.workunit.client.0.smithi016.stdout: "tunables": { 2020-03-11T22:44:46.781 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_tries": 0, 2020-03-11T22:44:46.782 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:44:46.782 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_total_tries": 50, 2020-03-11T22:44:46.782 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:44:46.782 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:44:46.782 INFO:tasks.workunit.client.0.smithi016.stdout: "chooseleaf_stable": 1, 2020-03-11T22:44:46.782 INFO:tasks.workunit.client.0.smithi016.stdout: "straw_calc_version": 1, 2020-03-11T22:44:46.783 INFO:tasks.workunit.client.0.smithi016.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:44:46.783 INFO:tasks.workunit.client.0.smithi016.stdout: "profile": "jewel", 2020-03-11T22:44:46.783 INFO:tasks.workunit.client.0.smithi016.stdout: "optimal_tunables": 1, 2020-03-11T22:44:46.783 INFO:tasks.workunit.client.0.smithi016.stdout: "legacy_tunables": 0, 2020-03-11T22:44:46.784 INFO:tasks.workunit.client.0.smithi016.stdout: "minimum_required_version": "jewel", 2020-03-11T22:44:46.784 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables": 1, 2020-03-11T22:44:46.784 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables2": 1, 2020-03-11T22:44:46.784 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v2_rules": 1, 2020-03-11T22:44:46.785 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables3": 1, 2020-03-11T22:44:46.785 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v3_rules": 0, 2020-03-11T22:44:46.785 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v4_buckets": 1, 2020-03-11T22:44:46.785 INFO:tasks.workunit.client.0.smithi016.stdout: "require_feature_tunables5": 1, 2020-03-11T22:44:46.786 INFO:tasks.workunit.client.0.smithi016.stdout: "has_v5_rules": 0 2020-03-11T22:44:46.786 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.786 INFO:tasks.workunit.client.0.smithi016.stdout: "choose_args": {} 2020-03-11T22:44:46.786 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.786 INFO:tasks.workunit.client.0.smithi016.stdout: "fsmap": { 2020-03-11T22:44:46.787 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:44:46.787 INFO:tasks.workunit.client.0.smithi016.stdout: "default_fscid": -1, 2020-03-11T22:44:46.787 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": { 2020-03-11T22:44:46.787 INFO:tasks.workunit.client.0.smithi016.stdout: "compat": {}, 2020-03-11T22:44:46.787 INFO:tasks.workunit.client.0.smithi016.stdout: "ro_compat": {}, 2020-03-11T22:44:46.788 INFO:tasks.workunit.client.0.smithi016.stdout: "incompat": { 2020-03-11T22:44:46.788 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_1": "base v0.20", 2020-03-11T22:44:46.788 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:44:46.788 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:44:46.788 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:44:46.789 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:44:46.789 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:44:46.789 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_8": "no anchor table", 2020-03-11T22:44:46.789 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_9": "file layout v2", 2020-03-11T22:44:46.790 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:44:46.790 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.790 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.790 INFO:tasks.workunit.client.0.smithi016.stdout: "feature_flags": { 2020-03-11T22:44:46.790 INFO:tasks.workunit.client.0.smithi016.stdout: "enable_multiple": false, 2020-03-11T22:44:46.791 INFO:tasks.workunit.client.0.smithi016.stdout: "ever_enabled_multiple": false 2020-03-11T22:44:46.791 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.791 INFO:tasks.workunit.client.0.smithi016.stdout: "standbys": [], 2020-03-11T22:44:46.791 INFO:tasks.workunit.client.0.smithi016.stdout: "filesystems": [] 2020-03-11T22:44:46.792 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.792 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:44:46.792 INFO:tasks.workunit.client.0.smithi016.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:44:46.792 INFO:tasks.workunit.client.0.smithi016.stdout: "auth": { 2020-03-11T22:44:46.792 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:44:46.793 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 5, 2020-03-11T22:44:46.793 INFO:tasks.workunit.client.0.smithi016.stdout: "num_secrets": 3 2020-03-11T22:44:46.793 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.793 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:44:46.793 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_active": 1, 2020-03-11T22:44:46.793 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_unknown": 0, 2020-03-11T22:44:46.794 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osd": 3, 2020-03-11T22:44:46.794 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_sum": { 2020-03-11T22:44:46.794 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:44:46.794 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48430, 2020-03-11T22:44:46.795 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 101, 2020-03-11T22:44:46.795 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 1, 2020-03-11T22:44:46.795 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 303, 2020-03-11T22:44:46.795 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:46.795 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:44:46.796 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:46.796 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:46.796 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:46.796 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:46.797 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:44:46.797 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:44:46.797 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:44:46.797 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 101, 2020-03-11T22:44:46.797 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 101, 2020-03-11T22:44:46.798 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:46.798 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:46.798 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:46.798 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:46.799 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:46.799 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:46.800 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:44:46.800 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:46.800 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:46.800 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:44:46.800 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:44:46.801 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:44:46.801 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:44:46.801 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:44:46.801 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:46.802 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:46.802 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:46.802 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:46.802 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:46.803 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:46.803 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:46.803 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:46.803 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:46.803 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:44:46.803 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:44:46.803 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.804 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:44:46.804 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:44:46.804 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:44:46.804 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:44:46.804 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 0, 2020-03-11T22:44:46.804 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 0, 2020-03-11T22:44:46.804 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:44:46.805 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:46.805 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:44:46.805 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:44:46.805 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:44:46.805 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.805 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 18, 2020-03-11T22:44:46.805 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:46.805 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 3, 2020-03-11T22:44:46.806 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 3, 2020-03-11T22:44:46.806 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 0 2020-03-11T22:44:46.806 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.806 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum": { 2020-03-11T22:44:46.806 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:44:46.806 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:44:46.806 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:44:46.807 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:44:46.807 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:46.807 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:46.807 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:44:46.807 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3181440, 2020-03-11T22:44:46.807 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 35392, 2020-03-11T22:44:46.807 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 9, 2020-03-11T22:44:46.807 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145718, 2020-03-11T22:44:46.808 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311391360, 2020-03-11T22:44:46.808 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:44:46.808 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:44:46.808 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318864752640, 2020-03-11T22:44:46.808 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:46.808 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 36241408, 2020-03-11T22:44:46.808 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 1562189, 2020-03-11T22:44:46.809 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:44:46.809 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:46.809 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:44:46.809 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 9484, 2020-03-11T22:44:46.809 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221215988 2020-03-11T22:44:46.809 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.809 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:44:46.810 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:46.810 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:46.810 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:46.810 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:44:46.810 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:44:46.810 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:44:46.811 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.811 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:44:46.811 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:46.811 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:46.811 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:46.811 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:44:46.811 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.812 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:44:46.812 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:44:46.812 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.812 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_sum_by_class": { 2020-03-11T22:44:46.812 INFO:tasks.workunit.client.0.smithi016.stdout: "hdd": { 2020-03-11T22:44:46.812 INFO:tasks.workunit.client.0.smithi016.stdout: "up_from": 0, 2020-03-11T22:44:46.812 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 0, 2020-03-11T22:44:46.812 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pgs": 3, 2020-03-11T22:44:46.813 INFO:tasks.workunit.client.0.smithi016.stdout: "num_osds": 3, 2020-03-11T22:44:46.813 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:46.813 INFO:tasks.workunit.client.0.smithi016.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:46.813 INFO:tasks.workunit.client.0.smithi016.stdout: "kb": 314572800, 2020-03-11T22:44:46.813 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used": 3181440, 2020-03-11T22:44:46.813 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_data": 35392, 2020-03-11T22:44:46.813 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_omap": 9, 2020-03-11T22:44:46.814 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_used_meta": 3145718, 2020-03-11T22:44:46.814 INFO:tasks.workunit.client.0.smithi016.stdout: "kb_avail": 311391360, 2020-03-11T22:44:46.814 INFO:tasks.workunit.client.0.smithi016.stdout: "statfs": { 2020-03-11T22:44:46.814 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 322122547200, 2020-03-11T22:44:46.814 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 318864752640, 2020-03-11T22:44:46.814 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:46.814 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 36241408, 2020-03-11T22:44:46.815 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 1562189, 2020-03-11T22:44:46.815 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:44:46.815 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:46.815 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:44:46.815 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 9484, 2020-03-11T22:44:46.815 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 3221215988 2020-03-11T22:44:46.815 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.815 INFO:tasks.workunit.client.0.smithi016.stdout: "hb_peers": [], 2020-03-11T22:44:46.816 INFO:tasks.workunit.client.0.smithi016.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:46.816 INFO:tasks.workunit.client.0.smithi016.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:46.816 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:46.816 INFO:tasks.workunit.client.0.smithi016.stdout: "op_queue_age_hist": { 2020-03-11T22:44:46.816 INFO:tasks.workunit.client.0.smithi016.stdout: "histogram": [], 2020-03-11T22:44:46.816 INFO:tasks.workunit.client.0.smithi016.stdout: "upper_bound": 1 2020-03-11T22:44:46.816 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.817 INFO:tasks.workunit.client.0.smithi016.stdout: "perf_stat": { 2020-03-11T22:44:46.817 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:46.817 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:46.817 INFO:tasks.workunit.client.0.smithi016.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:46.817 INFO:tasks.workunit.client.0.smithi016.stdout: "apply_latency_ns": 0 2020-03-11T22:44:46.817 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.818 INFO:tasks.workunit.client.0.smithi016.stdout: "alerts": [], 2020-03-11T22:44:46.818 INFO:tasks.workunit.client.0.smithi016.stdout: "network_ping_times": [] 2020-03-11T22:44:46.818 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.818 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.818 INFO:tasks.workunit.client.0.smithi016.stdout: "pool_stats": [ 2020-03-11T22:44:46.818 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.818 INFO:tasks.workunit.client.0.smithi016.stdout: "poolid": 1, 2020-03-11T22:44:46.819 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg": 1, 2020-03-11T22:44:46.819 INFO:tasks.workunit.client.0.smithi016.stdout: "stat_sum": { 2020-03-11T22:44:46.819 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes": 48430, 2020-03-11T22:44:46.819 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects": 101, 2020-03-11T22:44:46.819 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_clones": 1, 2020-03-11T22:44:46.819 INFO:tasks.workunit.client.0.smithi016.stdout: "num_object_copies": 303, 2020-03-11T22:44:46.819 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:46.819 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_missing": 0, 2020-03-11T22:44:46.820 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:46.820 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:46.820 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:46.820 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:46.820 INFO:tasks.workunit.client.0.smithi016.stdout: "num_whiteouts": 0, 2020-03-11T22:44:46.820 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read": 0, 2020-03-11T22:44:46.820 INFO:tasks.workunit.client.0.smithi016.stdout: "num_read_kb": 0, 2020-03-11T22:44:46.821 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write": 101, 2020-03-11T22:44:46.821 INFO:tasks.workunit.client.0.smithi016.stdout: "num_write_kb": 101, 2020-03-11T22:44:46.821 INFO:tasks.workunit.client.0.smithi016.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:46.821 INFO:tasks.workunit.client.0.smithi016.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:46.821 INFO:tasks.workunit.client.0.smithi016.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:46.821 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:46.821 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:46.821 INFO:tasks.workunit.client.0.smithi016.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:46.822 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_omap": 0, 2020-03-11T22:44:46.822 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:46.822 INFO:tasks.workunit.client.0.smithi016.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:46.822 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush": 0, 2020-03-11T22:44:46.822 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_kb": 0, 2020-03-11T22:44:46.822 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict": 0, 2020-03-11T22:44:46.822 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_kb": 0, 2020-03-11T22:44:46.823 INFO:tasks.workunit.client.0.smithi016.stdout: "num_promote": 0, 2020-03-11T22:44:46.823 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:46.823 INFO:tasks.workunit.client.0.smithi016.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:46.823 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:46.823 INFO:tasks.workunit.client.0.smithi016.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:46.823 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:46.823 INFO:tasks.workunit.client.0.smithi016.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:46.823 INFO:tasks.workunit.client.0.smithi016.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:46.824 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:46.824 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:46.824 INFO:tasks.workunit.client.0.smithi016.stdout: "num_omap_keys": 0, 2020-03-11T22:44:46.824 INFO:tasks.workunit.client.0.smithi016.stdout: "num_objects_repaired": 0 2020-03-11T22:44:46.824 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.824 INFO:tasks.workunit.client.0.smithi016.stdout: "store_stats": { 2020-03-11T22:44:46.824 INFO:tasks.workunit.client.0.smithi016.stdout: "total": 0, 2020-03-11T22:44:46.825 INFO:tasks.workunit.client.0.smithi016.stdout: "available": 0, 2020-03-11T22:44:46.825 INFO:tasks.workunit.client.0.smithi016.stdout: "internally_reserved": 0, 2020-03-11T22:44:46.825 INFO:tasks.workunit.client.0.smithi016.stdout: "allocated": 19857408, 2020-03-11T22:44:46.825 INFO:tasks.workunit.client.0.smithi016.stdout: "data_stored": 1241088, 2020-03-11T22:44:46.825 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed": 0, 2020-03-11T22:44:46.825 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:46.825 INFO:tasks.workunit.client.0.smithi016.stdout: "data_compressed_original": 0, 2020-03-11T22:44:46.826 INFO:tasks.workunit.client.0.smithi016.stdout: "omap_allocated": 0, 2020-03-11T22:44:46.826 INFO:tasks.workunit.client.0.smithi016.stdout: "internal_metadata": 0 2020-03-11T22:44:46.826 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.826 INFO:tasks.workunit.client.0.smithi016.stdout: "log_size": 18, 2020-03-11T22:44:46.826 INFO:tasks.workunit.client.0.smithi016.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:46.826 INFO:tasks.workunit.client.0.smithi016.stdout: "up": 3, 2020-03-11T22:44:46.826 INFO:tasks.workunit.client.0.smithi016.stdout: "acting": 3, 2020-03-11T22:44:46.827 INFO:tasks.workunit.client.0.smithi016.stdout: "num_store_stats": 3 2020-03-11T22:44:46.827 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.827 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.827 INFO:tasks.workunit.client.0.smithi016.stdout: "osd_stats": [ 2020-03-11T22:44:46.827 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.827 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:44:46.827 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 180388626442 2020-03-11T22:44:46.827 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.828 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.828 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:44:46.828 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 42949672979 2020-03-11T22:44:46.828 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.828 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.828 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:44:46.828 INFO:tasks.workunit.client.0.smithi016.stdout: "seq": 193273528330 2020-03-11T22:44:46.828 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.829 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.829 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_state": [ 2020-03-11T22:44:46.829 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.829 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "active+undersized+degraded", 2020-03-11T22:44:46.829 INFO:tasks.workunit.client.0.smithi016.stdout: "num": 1 2020-03-11T22:44:46.829 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.829 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.830 INFO:tasks.workunit.client.0.smithi016.stdout: "num_pg_by_osd": [ 2020-03-11T22:44:46.830 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.830 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2, 2020-03-11T22:44:46.830 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:44:46.830 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:44:46.830 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:46.830 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.831 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.831 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 2147483647, 2020-03-11T22:44:46.831 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:44:46.831 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:44:46.831 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:46.831 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.831 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.832 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 1, 2020-03-11T22:44:46.832 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 0, 2020-03-11T22:44:46.832 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 0, 2020-03-11T22:44:46.832 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:46.832 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.832 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.832 INFO:tasks.workunit.client.0.smithi016.stdout: "osd": 0, 2020-03-11T22:44:46.833 INFO:tasks.workunit.client.0.smithi016.stdout: "num_primary_pg": 1, 2020-03-11T22:44:46.833 INFO:tasks.workunit.client.0.smithi016.stdout: "num_acting_pg": 1, 2020-03-11T22:44:46.833 INFO:tasks.workunit.client.0.smithi016.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:46.833 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.833 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.833 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": [ 2020-03-11T22:44:46.833 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:44:46.834 INFO:tasks.workunit.client.0.smithi016.stdout: "pool": 1, 2020-03-11T22:44:46.834 INFO:tasks.workunit.client.0.smithi016.stdout: "purged_snaps": {} 2020-03-11T22:44:46.834 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.834 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:44:46.834 INFO:tasks.workunit.client.0.smithi016.stdout: "servicemap": { 2020-03-11T22:44:46.834 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": 1, 2020-03-11T22:44:46.834 INFO:tasks.workunit.client.0.smithi016.stdout: "modified": "2020-03-11T22:42:11.853755+0000", 2020-03-11T22:44:46.835 INFO:tasks.workunit.client.0.smithi016.stdout: "services": {} 2020-03-11T22:44:46.835 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:44:46.835 INFO:tasks.workunit.client.0.smithi016.stdout: "paxos": { 2020-03-11T22:44:46.835 INFO:tasks.workunit.client.0.smithi016.stdout: "first_committed": 1, 2020-03-11T22:44:46.835 INFO:tasks.workunit.client.0.smithi016.stdout: "last_committed": 436, 2020-03-11T22:44:46.835 INFO:tasks.workunit.client.0.smithi016.stdout: "last_pn": 0, 2020-03-11T22:44:46.835 INFO:tasks.workunit.client.0.smithi016.stdout: "accepted_pn": 0 2020-03-11T22:44:46.835 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:44:46.836 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:44:46.836 INFO:tasks.workunit.client.0.smithi016.stderr:report 1282704038 2020-03-11T22:44:46.837 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:44:46.837 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-03-11T22:44:46.837 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:46.837 INFO:tasks.workunit.client.0.smithi016.stdout:delay recovery 2020-03-11T22:44:46.988 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:46.988 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:09.765036+0000 40'102 46:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:44:46.988 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:44:46.988 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:46.990 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:44:46.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:46.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:44:46.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:44:46.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:44:46.999 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:47.000 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:47.000 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:44:47.000 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.0.asok 2020-03-11T22:44:47.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:47.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:44:47.096 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:47.104 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:47.105 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:44:47.105 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:44:47.105 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:44:47.105 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:47.105 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:47.105 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:44:47.105 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.2.asok 2020-03-11T22:44:47.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:47.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:44:47.202 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:47.209 INFO:tasks.workunit.client.0.smithi016.stdout:revive divergent 1 2020-03-11T22:44:47.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-03-11T22:44:47.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:47.553 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:47.553 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:09.765036+0000 40'102 46:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:44:47.553 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:44:47.553 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:47.555 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:44:47.563 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-03-11T22:44:48.127 INFO:tasks.workunit.client.0.smithi016.stderr:noup is set 2020-03-11T22:44:48.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:44:48.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:48.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:48.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:44:48.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:48.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:44:48.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:48.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:48.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:48.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:48.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:44:48.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:44:48.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:48.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:48.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:48.141 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:48.141 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:48.141 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:48.141 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:48.142 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:48.142 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:44:48.142 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:44:48.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:44:48.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:48.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:48.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:48.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:48.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:48.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:48.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:48.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:48.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:44:48.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:44:48.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:48.144 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:44:48.144 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:44:48.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:44:48.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:48.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:48.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:48.166 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:48.163+0000 7efffa61cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:48.168 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:48.164+0000 7efffa61cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:48.170 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:48.167+0000 7efffa61cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:44:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-03-11T22:44:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:48.687+0000 7efffa61cec0 -1 Falling back to public interface 2020-03-11T22:44:49.519 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:44:49.515+0000 7efffa61cec0 -1 osd.1 30 log_to_monitors {default=true} 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi016.stdout:delay recovery divergent 2020-03-11T22:44:53.868 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:53.868 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:09.765036+0000 40'102 46:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:44:53.868 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:44:53.868 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:53.869 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:44:53.879 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-03-11T22:44:53.879 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:44:53.879 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:44:53.879 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:53.879 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:53.879 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:44:53.880 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17411/ceph-osd.1.asok 2020-03-11T22:44:53.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:53.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17411/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:44:53.976 INFO:tasks.workunit.client.0.smithi016.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:53.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-03-11T22:44:54.724 INFO:tasks.workunit.client.0.smithi016.stderr:noup is unset 2020-03-11T22:44:54.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-03-11T22:44:54.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:54.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:54.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:54.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:54.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:54.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:54.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:54.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:54.736 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:44:55.156 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 42 up_thru 45 down_at 41 last_clean_interval [36,40) [v2:127.0.0.1:6802/50346,v1:127.0.0.1:6803/50346] [v2:127.0.0.1:6804/50346,v1:127.0.0.1:6805/50346] exists,up 6c73bd18-95db-4897-9cb7-6ae241013376 2020-03-11T22:44:55.157 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:44:55.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:55.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:55.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:55.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-03-11T22:44:55.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:55.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:44:55.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:55.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:55.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:55.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:55.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:55.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:55.531 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/51993,v1:127.0.0.1:6819/51993] [v2:127.0.0.1:6820/51993,v1:127.0.0.1:6821/51993] exists,up 6e1b8749-4a8a-4a6e-8118-60aab81e6f12 2020-03-11T22:44:55.531 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:44:55.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:55.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:55.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:55.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-03-11T22:44:55.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:55.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:55.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:55.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:55.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:55.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:55.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:55.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:55.938 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 45 up_thru 0 down_at 44 last_clean_interval [37,40) [v2:127.0.0.1:6810/50578,v1:127.0.0.1:6811/50578] [v2:127.0.0.1:6812/50578,v1:127.0.0.1:6813/50578] exists,up 6d353a96-21d2-42eb-bd15-df39bc2f3fbb 2020-03-11T22:44:55.938 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:55.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:55.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:55.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:56.281 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:56.281 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:09.765036+0000 40'102 46:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:44:56.281 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:44:56.281 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:56.283 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:44:56.292 INFO:tasks.workunit.client.0.smithi016.stdout:wait for peering 2020-03-11T22:44:56.293 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-03-11T22:44:56.293 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:56.631 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:56.631 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:09.765036+0000 40'102 46:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:44:56.631 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:44:56.631 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:56.632 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:44:56.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-03-11T22:45:05.800 INFO:tasks.workunit.client.0.smithi016.stdout:killing divergent 1 2020-03-11T22:45:05.801 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-03-11T22:45:05.801 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:06.153 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:06.153 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:44:55.040663+0000 40'102 52:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:45:06.153 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:45:06.154 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:06.155 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:45:06.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:45:06.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:06.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:06.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:06.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:06.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:06.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:06.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-03-11T22:45:06.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:06.270 INFO:tasks.workunit.client.0.smithi016.stdout:reviving divergent 1 2020-03-11T22:45:06.613 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:06.613 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:44:55.040663+0000 40'102 52:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:45:06.613 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:45:06.613 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:06.614 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:45:06.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:45:06.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:06.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:06.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:45:06.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:06.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:45:06.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:06.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:06.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:06.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:06.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:45:06.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:45:06.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:06.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:06.628 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:06.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:06.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:06.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:06.628 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:06.628 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:06.628 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:45:06.629 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:45:06.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' 2020-03-11T22:45:06.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:06.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:06.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:06.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:06.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:06.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:06.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:06.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:06.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:45:06.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:45:06.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7572dbb-081c-44c5-977c-66ff16469e9f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17411/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:06.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:45:06.631 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:45:06.631 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:45:06.631 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:06.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:06.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:06.653 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:06.650+0000 7fe7f8ea1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:06.655 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:06.652+0000 7fe7f8ea1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:06.656 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:06.653+0000 7fe7f8ea1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:06.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:45:06.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:06.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:06.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:06.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:06.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:06.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:06.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:06.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:06.991 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:45:07.174 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:07.170+0000 7fe7f8ea1ec0 -1 Falling back to public interface 2020-03-11T22:45:07.357 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/51993,v1:127.0.0.1:6819/51993] [v2:127.0.0.1:6820/51993,v1:127.0.0.1:6821/51993] exists,up 6e1b8749-4a8a-4a6e-8118-60aab81e6f12 2020-03-11T22:45:07.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:07.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:07.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:07.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-03-11T22:45:08.007 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:08.004+0000 7fe7f8ea1ec0 -1 osd.1 52 log_to_monitors {default=true} 2020-03-11T22:45:27.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-03-11T22:45:27.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:27.359 INFO:tasks.workunit.client.0.smithi016.stdout:allowing recovery 2020-03-11T22:45:27.707 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:27.707 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:45:09.344025+0000 52'103 55:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:45:27.708 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:45:27.708 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:27.709 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:45:27.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:27.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:45:27.845 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:27.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:27.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:45:27.981 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:27.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:27.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:45:28.117 INFO:tasks.workunit.client.0.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:28.125 INFO:tasks.workunit.client.0.smithi016.stdout:reading divergent objects 2020-03-11T22:45:28.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-03-11T22:45:28.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:28.487 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:28.487 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:45:09.344025+0000 52'103 55:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:34.947690+0000 0'0 2020-03-11T22:42:34.947690+0000 0 2020-03-11T22:45:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:45:28.487 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:28.488 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:45:28.497 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:45:28.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-03-11T22:45:28.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:28.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:45:28.541 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:28.541 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:45:28.571 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:28.571 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:45:28.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:28.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:45:28.630 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-03-11T22:45:28.631 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:28.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:28.636 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:42:56.829267+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.636 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:42:56.829267+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.636 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:28.636 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-03-11T22:45:28.637 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-03-11T22:45:28.637 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-03-11T22:45:28.637 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-03-11T22:43:03.849840+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.637 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-03-11T22:43:03.849840+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.637 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:28.637 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-03-11T22:45:28.638 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-03-11T22:45:28.638 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:28.638 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2020-03-11T22:45:28.638 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-03-11T22:43:03.849840+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.638 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-03-11T22:45:28.638 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4546.0:1 2020-03-11T22:43:02.655446+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-03-11T22:43:03.662124+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.639 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4546.0:1 2020-03-11T22:43:02.655446+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.639 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-03-11T22:43:03.662124+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.639 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:28.639 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2020-03-11T22:45:28.639 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:45:28.640 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:28.640 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2020-03-11T22:45:28.640 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-03-11T22:43:03.662124+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:28.640 INFO:tasks.workunit.client.0.smithi016.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:54.928+0000 7effd87d7700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:45:28.640 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:28.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:28.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-03-11T22:45:28.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-03-11T22:45:28.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:45:28.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:45:28.643 INFO:tasks.workunit.client.0.smithi016.stdout:success 2020-03-11T22:45:29.156 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' does not exist 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:29.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:29.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:29.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:29.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:29.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:29.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:45:29.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:45:29.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:29.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:45:29.274 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:29.274 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:29.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:29.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:29.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:29.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:29.279 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:29.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:29.280 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:29.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:29.280 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:29.281 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:29.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:29.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:29.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:29.283 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:29.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:29.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:29.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:45:29.298 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:29.298 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:29.298 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:45:29.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:45:29.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:29.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:29.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:45:29.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/divergent-priors 0 2020-03-11T22:45:29.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:45:29.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:45:29.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:45:29.301 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:29.301 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:29.301 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:29.301 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:29.301 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:29.302 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:29.302 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:29.302 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:29.303 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:29.303 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:29.303 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:29.304 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:29.305 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:29.305 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:29.306 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:29.306 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:29.306 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:29.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:45:29.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:45:29.308 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:29.308 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:29.309 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17411 2020-03-11T22:45:29.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17411 2020-03-11T22:45:29.310 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:29.310 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:29.310 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:45:29.310 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:45:29.311 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:45:29.379 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-03-11T22:45:29.379 INFO:teuthology.orchestra.run.smithi016:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-03-11T22:45:29.413 INFO:tasks.workunit.client.0.smithi016.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:45:29.421 INFO:tasks.workunit.client.0.smithi016.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:45:29.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:45:29.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:45:29.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:45:29.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:45:29.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:45:29.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:45:29.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:45:29.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:45:29.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/ec-error-rollforward 2020-03-11T22:45:29.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-03-11T22:45:29.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-03-11T22:45:29.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-03-11T22:45:29.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-03-11T22:45:29.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-03-11T22:45:29.423 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-03-11T22:45:29.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none ' 2020-03-11T22:45:29.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:29.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-03-11T22:45:29.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-03-11T22:45:29.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-03-11T22:45:29.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-03-11T22:45:29.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-03-11T22:45:29.425 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-03-11T22:45:29.425 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-03-11T22:45:29.425 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:45:29.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-03-11T22:45:29.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-03-11T22:45:29.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-03-11T22:45:29.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-03-11T22:45:29.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-03-11T22:45:29.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:45:29.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:29.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:45:29.428 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:29.428 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:29.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:29.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:29.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:29.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:29.429 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:29.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:29.429 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:29.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:29.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:29.430 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:29.431 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:29.431 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:29.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:29.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:29.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:29.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:29.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:45:29.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:29.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:29.433 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:29.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52921 2020-03-11T22:45:29.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:29.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:29.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-03-11T22:45:29.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:45:29.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:29.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:29.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52921 2020-03-11T22:45:29.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:45:29.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:45:29.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:45:29.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:45:29.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-03-11T22:45:29.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-03-11T22:45:29.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-03-11T22:45:29.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-03-11T22:45:29.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-03-11T22:45:29.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:45:29.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:45:29.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:45:29.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-03-11T22:45:29.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-03-11T22:45:29.732 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:45:29.732 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:29.733 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:29.733 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:29.733 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:29.733 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:29.733 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:29.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:45:29.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:45:29.935 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:45:29.935 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:29.935 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:29.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:45:29.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:45:29.936 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:29.936 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:29.936 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:29.936 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:29.936 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:29.937 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:29.937 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52921/ceph-mon.a.asok 2020-03-11T22:45:29.937 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:29.937 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52921/ceph-mon.a.asok config get fsid 2020-03-11T22:45:30.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:45:30.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:30.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:30.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:45:30.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:45:30.029 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:30.029 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:30.029 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:30.029 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:30.029 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:30.029 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:30.030 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52921/ceph-mon.a.asok 2020-03-11T22:45:30.030 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:30.030 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52921/ceph-mon.a.asok config get mon_host 2020-03-11T22:45:30.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-03-11T22:45:30.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-03-11T22:45:30.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:45:30.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:45:30.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:45:30.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-03-11T22:45:30.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:45:31.027 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:45:31.027 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:31.027 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:31.028 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:31.028 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:31.028 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:31.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:31.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:31.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:31.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-03-11T22:45:31.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:31.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:31.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:45:31.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:31.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:45:31.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:31.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:31.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:31.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:31.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:31.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:31.054 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:31.054 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:31.054 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:31.054 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:31.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:31.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:31.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:31.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:31.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:31.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:31.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:31.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:31.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:31.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:31.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:31.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:45:31.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:31.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0017f8c9-f5d1-481e-8b49-30ba59a3023b 2020-03-11T22:45:31.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 0017f8c9-f5d1-481e-8b49-30ba59a3023b' 2020-03-11T22:45:31.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:31.057 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 0017f8c9-f5d1-481e-8b49-30ba59a3023b 2020-03-11T22:45:31.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQALamletreuAxAAM7VoPFt4h+SW27BprDa8WQ== 2020-03-11T22:45:31.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQALamletreuAxAAM7VoPFt4h+SW27BprDa8WQ=="}' 2020-03-11T22:45:31.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0017f8c9-f5d1-481e-8b49-30ba59a3023b -i td/ec-error-rollforward/0/new.json 2020-03-11T22:45:31.433 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:45:31.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-03-11T22:45:31.443 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQALamletreuAxAAM7VoPFt4h+SW27BprDa8WQ== --osd-uuid 0017f8c9-f5d1-481e-8b49-30ba59a3023b 2020-03-11T22:45:31.464 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:31.462+0000 7f9a0d407ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:31.466 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:31.463+0000 7f9a0d407ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:31.468 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:31.465+0000 7f9a0d407ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:31.468 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:31.465+0000 7f9a0d407ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:31.468 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:31.465+0000 7f9a0d407ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:31.468 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:31.465+0000 7f9a0d407ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:31.516 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:31.513+0000 7f9a0d407ec0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-03-11T22:45:34.577 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-03-11T22:45:34.577 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:34.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:45:34.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:34.578 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:45:34.837 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:45:34.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:45:34.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:34.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:34.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:34.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:34.856 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:34.853+0000 7fbbe2ee7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:34.857 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:34.855+0000 7fbbe2ee7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:34.859 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:34.857+0000 7fbbe2ee7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:35.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:45:35.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:35.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:35.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:35.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:35.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:35.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:35.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:35.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:35.213 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:45:35.389 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:35.386+0000 7fbbe2ee7ec0 -1 Falling back to public interface 2020-03-11T22:45:35.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:36.157 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:36.155+0000 7fbbe2ee7ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:45:36.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:36.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:36.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:36.616 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:36.617 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:36.617 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:45:37.017 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53350,v1:127.0.0.1:6803/53350] [v2:127.0.0.1:6804/53350,v1:127.0.0.1:6805/53350] exists,up 0017f8c9-f5d1-481e-8b49-30ba59a3023b 2020-03-11T22:45:37.018 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 99f0bb56-e5bb-479d-aa45-be21d7c8aa46 2020-03-11T22:45:37.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:37.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:37.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:37.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-03-11T22:45:37.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:45:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:45:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:37.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:37.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:45:37.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:45:37.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:37.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:37.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:37.020 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:37.020 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:37.021 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:37.021 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:37.021 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:37.021 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:37.021 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:37.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:37.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:37.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:37.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:37.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:37.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:37.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:37.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:37.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:37.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:37.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:37.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:45:37.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:37.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=99f0bb56-e5bb-479d-aa45-be21d7c8aa46 2020-03-11T22:45:37.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 99f0bb56-e5bb-479d-aa45-be21d7c8aa46' 2020-03-11T22:45:37.023 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:37.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQARamleNsGcARAAKAtTsDsx4sefdOXH5w6B8A== 2020-03-11T22:45:37.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQARamleNsGcARAAKAtTsDsx4sefdOXH5w6B8A=="}' 2020-03-11T22:45:37.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 99f0bb56-e5bb-479d-aa45-be21d7c8aa46 -i td/ec-error-rollforward/1/new.json 2020-03-11T22:45:37.467 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:45:37.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-03-11T22:45:37.478 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQARamleNsGcARAAKAtTsDsx4sefdOXH5w6B8A== --osd-uuid 99f0bb56-e5bb-479d-aa45-be21d7c8aa46 2020-03-11T22:45:37.499 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:37.496+0000 7f89a9566ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:37.522 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:37.519+0000 7f89a9566ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:37.524 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:37.521+0000 7f89a9566ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:37.525 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:37.521+0000 7f89a9566ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:37.525 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:37.521+0000 7f89a9566ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:37.525 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:37.521+0000 7f89a9566ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:37.615 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:37.611+0000 7f89a9566ec0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-03-11T22:45:40.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-03-11T22:45:40.798 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:40.800 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:45:40.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:45:40.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:41.293 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:45:41.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:45:41.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:41.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:41.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:41.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:41.312 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:41.308+0000 7fa4845b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:41.314 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:41.310+0000 7fa4845b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:41.315 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:41.312+0000 7fa4845b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:41.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:45:41.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:41.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:41.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:41.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:41.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:41.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:41.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:41.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:41.682 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:45:41.834 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:41.830+0000 7fa4845b1ec0 -1 Falling back to public interface 2020-03-11T22:45:42.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:42.647 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:42.639+0000 7fa4845b1ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:45:43.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:43.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:43.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:43.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:43.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:43.039 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:45:43.425 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53705,v1:127.0.0.1:6811/53705] [v2:127.0.0.1:6812/53705,v1:127.0.0.1:6813/53705] exists,up 99f0bb56-e5bb-479d-aa45-be21d7c8aa46 2020-03-11T22:45:43.425 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:43.425 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:43.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:43.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-03-11T22:45:43.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:43.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:43.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:45:43.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:43.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:45:43.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:43.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:43.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:43.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:43.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:45:43.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:45:43.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:43.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:43.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:43.428 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:43.428 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:43.428 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:43.428 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:43.429 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:43.429 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:43.429 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:43.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:43.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:43.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:43.429 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:43.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:43.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:43.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:43.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:43.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:43.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:43.431 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:43.431 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:45:43.431 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:43.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=04a7ddfe-20bc-4bf5-93a4-a8609fc50fe8 2020-03-11T22:45:43.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 04a7ddfe-20bc-4bf5-93a4-a8609fc50fe8' 2020-03-11T22:45:43.432 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:43.432 INFO:tasks.workunit.client.0.smithi016.stdout:add osd2 04a7ddfe-20bc-4bf5-93a4-a8609fc50fe8 2020-03-11T22:45:43.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAXamleayYLGhAAXBt/5Aii+SUeV3XWPM2iIw== 2020-03-11T22:45:43.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAXamleayYLGhAAXBt/5Aii+SUeV3XWPM2iIw=="}' 2020-03-11T22:45:43.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 04a7ddfe-20bc-4bf5-93a4-a8609fc50fe8 -i td/ec-error-rollforward/2/new.json 2020-03-11T22:45:44.012 INFO:tasks.workunit.client.0.smithi016.stdout:2 2020-03-11T22:45:44.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-03-11T22:45:44.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAXamleayYLGhAAXBt/5Aii+SUeV3XWPM2iIw== --osd-uuid 04a7ddfe-20bc-4bf5-93a4-a8609fc50fe8 2020-03-11T22:45:44.043 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:44.040+0000 7f1ab859fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:44.045 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:44.042+0000 7f1ab859fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:44.046 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:44.043+0000 7f1ab859fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:44.047 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:44.044+0000 7f1ab859fec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:44.047 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:44.044+0000 7f1ab859fec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:44.048 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:44.044+0000 7f1ab859fec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:44.146 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:44.143+0000 7f1ab859fec0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-03-11T22:45:47.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-03-11T22:45:47.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:47.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:45:47.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:47.274 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd2 key to auth repository 2020-03-11T22:45:47.760 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:45:47.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:45:47.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:47.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:47.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:47.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:47.778 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:47.775+0000 7fa844851ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:47.780 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:47.776+0000 7fa844851ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:47.782 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:47.778+0000 7fa844851ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:48.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:45:48.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:48.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:48.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:48.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:48.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:48.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:48.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:48.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:48.125 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:45:48.297 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:48.293+0000 7fa844851ec0 -1 Falling back to public interface 2020-03-11T22:45:48.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:49.079 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:49.076+0000 7fa844851ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:45:49.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:49.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:49.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:49.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:49.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:49.536 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:45:49.917 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54060,v1:127.0.0.1:6819/54060] [v2:127.0.0.1:6820/54060,v1:127.0.0.1:6821/54060] exists,up 04a7ddfe-20bc-4bf5-93a4-a8609fc50fe8 2020-03-11T22:45:49.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:49.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:49.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:49.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-03-11T22:45:49.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:49.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:49.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:45:49.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:49.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:45:49.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:49.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:49.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:49.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:49.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:45:49.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:45:49.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:49.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:49.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:49.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:49.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:49.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:49.920 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:49.920 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:49.920 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:45:49.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:49.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:45:49.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:49.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:49.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:49.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:49.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:49.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:49.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:49.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:49.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:49.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:49.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:45:49.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:49.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e06624c8-444b-4300-9b2a-6571f16cacac 2020-03-11T22:45:49.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 e06624c8-444b-4300-9b2a-6571f16cacac' 2020-03-11T22:45:49.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:49.924 INFO:tasks.workunit.client.0.smithi016.stdout:add osd3 e06624c8-444b-4300-9b2a-6571f16cacac 2020-03-11T22:45:49.931 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAdamleAv5aNxAAtgJgZOqMxLFD3CmN/TJhjQ== 2020-03-11T22:45:49.931 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAdamleAv5aNxAAtgJgZOqMxLFD3CmN/TJhjQ=="}' 2020-03-11T22:45:49.931 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e06624c8-444b-4300-9b2a-6571f16cacac -i td/ec-error-rollforward/3/new.json 2020-03-11T22:45:50.417 INFO:tasks.workunit.client.0.smithi016.stdout:3 2020-03-11T22:45:50.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-03-11T22:45:50.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAdamleAv5aNxAAtgJgZOqMxLFD3CmN/TJhjQ== --osd-uuid e06624c8-444b-4300-9b2a-6571f16cacac 2020-03-11T22:45:50.450 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:50.446+0000 7fcebb474ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:50.451 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:50.448+0000 7fcebb474ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:50.453 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:50.450+0000 7fcebb474ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:50.454 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:50.451+0000 7fcebb474ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:50.454 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:50.451+0000 7fcebb474ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:50.454 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:50.451+0000 7fcebb474ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:50.486 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:50.483+0000 7fcebb474ec0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-03-11T22:45:53.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-03-11T22:45:53.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:53.848 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd3 key to auth repository 2020-03-11T22:45:53.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:45:53.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:54.337 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.3 2020-03-11T22:45:54.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:45:54.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:54.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:54.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:54.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:54.356 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:54.353+0000 7f0190158ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:54.358 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:54.355+0000 7f0190158ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:54.359 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:54.356+0000 7f0190158ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:54.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:45:54.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:54.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:45:54.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:54.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:54.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:54.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:54.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:54.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:45:54.714 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:45:54.879 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:54.875+0000 7f0190158ec0 -1 Falling back to public interface 2020-03-11T22:45:55.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:55.654 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:45:55.651+0000 7f0190158ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:45:56.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:56.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:56.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:56.109 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:56.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:45:56.110 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:45:56.486 INFO:tasks.workunit.client.0.smithi016.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/54415,v1:127.0.0.1:6827/54415] [v2:127.0.0.1:6828/54415,v1:127.0.0.1:6829/54415] exists,up e06624c8-444b-4300-9b2a-6571f16cacac 2020-03-11T22:45:56.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:56.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:56.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:56.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-03-11T22:45:57.098 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-03-11T22:45:57.807 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'ec' already exists 2020-03-11T22:45:57.815 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-03-11T22:45:59.486 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-03-11T22:45:59.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 54060 2020-03-11T22:45:59.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=54686 2020-03-11T22:45:59.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:45:59.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-03-11T22:46:00.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 54709' 2020-03-11T22:46:00.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-03-11T22:46:00.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 54710' 2020-03-11T22:46:00.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-03-11T22:46:00.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 54711' 2020-03-11T22:46:00.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:46:00.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-03-11T22:46:01.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-03-11T22:46:01.492 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:01.492 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:01.492 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:01.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:01.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:01.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:01.602 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 54686 54709 54710 54711 2020-03-11T22:46:01.602 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-03-11T22:46:01.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-03-11T22:46:01.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:01.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:01.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:46:01.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:01.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:46:01.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:01.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:01.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:01.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:01.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:46:01.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:46:01.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:01.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:01.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:01.608 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:01.608 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:01.608 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:01.608 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:01.608 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:01.609 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:46:01.609 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:46:01.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:46:01.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:01.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:01.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:01.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:01.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:01.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:01.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:01.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:01.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:46:01.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:46:01.610 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:46:01.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:01.611 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-03-11T22:46:01.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:46:01.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:01.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:01.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:01.627 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:01.624+0000 7efcaccd8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:01.630 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:01.627+0000 7efcaccd8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:01.632 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:01.629+0000 7efcaccd8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:01.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:46:01.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:01.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:01.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:01.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:01.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:01.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:01.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:01.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:01.980 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:46:02.148 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:02.144+0000 7efcaccd8ec0 -1 Falling back to public interface 2020-03-11T22:46:02.341 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53350,v1:127.0.0.1:6803/53350] [v2:127.0.0.1:6804/53350,v1:127.0.0.1:6805/53350] exists,up 0017f8c9-f5d1-481e-8b49-30ba59a3023b 2020-03-11T22:46:02.341 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:46:02.342 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:02.342 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:02.342 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:02.342 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-03-11T22:46:02.342 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:02.342 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:02.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:46:02.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:02.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:46:02.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:02.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:02.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:02.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:02.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:46:02.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:46:02.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:02.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:02.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:02.344 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:02.344 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:02.345 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:02.345 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:02.345 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:02.345 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:46:02.345 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:46:02.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:46:02.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:02.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:02.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:02.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:02.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:02.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:02.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:02.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:02.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:46:02.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:46:02.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:02.347 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-03-11T22:46:02.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:46:02.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:02.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:02.348 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:02.358 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:02.355+0000 7fe6970d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:02.360 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:02.357+0000 7fe6970d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:02.361 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:02.358+0000 7fe6970d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:02.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:46:02.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:02.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:02.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:02.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:02.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:02.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:02.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:02.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:02.759 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:46:02.877 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:02.873+0000 7fe6970d3ec0 -1 Falling back to public interface 2020-03-11T22:46:03.046 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:03.043+0000 7efcaccd8ec0 -1 osd.0 26 log_to_monitors {default=true} 2020-03-11T22:46:03.129 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53705,v1:127.0.0.1:6811/53705] [v2:127.0.0.1:6812/53705,v1:127.0.0.1:6813/53705] exists,up 99f0bb56-e5bb-479d-aa45-be21d7c8aa46 2020-03-11T22:46:03.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:03.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:03.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:03.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-03-11T22:46:03.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:03.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:03.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:46:03.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:03.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:46:03.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:03.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:03.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:03.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:03.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:46:03.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:46:03.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:03.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:03.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:03.132 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:03.132 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:03.132 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:03.132 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:03.133 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:03.133 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:46:03.133 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:46:03.133 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:46:03.133 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:03.133 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:03.133 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:03.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:03.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:03.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:03.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:03.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:03.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:46:03.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:46:03.136 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:03.136 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-03-11T22:46:03.136 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:46:03.136 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:03.136 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:03.136 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:03.136 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:46:03.149 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:03.146+0000 7f2d0a04fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:03.154 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:03.151+0000 7f2d0a04fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:03.155 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:03.152+0000 7f2d0a04fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:03.671 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:03.668+0000 7f2d0a04fec0 -1 Falling back to public interface 2020-03-11T22:46:03.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:46:03.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:03.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:46:03.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:03.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:03.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:03.835 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:03.835 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:03.835 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:03.835 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:46:04.272 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:04.269+0000 7fe6970d3ec0 -1 osd.1 26 log_to_monitors {default=true} 2020-03-11T22:46:04.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:04.785 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:04.781+0000 7f2d0a04fec0 -1 osd.2 25 log_to_monitors {default=true} 2020-03-11T22:46:05.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:05.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:05.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:05.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:05.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:05.361 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:46:05.744 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 29 last_clean_interval [15,25) [v2:127.0.0.1:6818/55074,v1:127.0.0.1:6819/55074] [v2:127.0.0.1:6820/55074,v1:127.0.0.1:6821/55074] exists,up 04a7ddfe-20bc-4bf5-93a4-a8609fc50fe8 2020-03-11T22:46:05.744 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.3 2020-03-11T22:46:05.744 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:05.745 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:05.745 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:05.745 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-03-11T22:46:05.745 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:05.745 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:05.745 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-03-11T22:46:05.745 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:05.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:46:05.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:05.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:05.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:05.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:05.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:46:05.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:46:05.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:05.747 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:05.747 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:05.747 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:05.747 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:05.747 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:05.747 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:05.748 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:05.748 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:46:05.748 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:46:05.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' 2020-03-11T22:46:05.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:05.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:05.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:05.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:05.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:05.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:05.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:05.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:05.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:46:05.750 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-03-11T22:46:05.750 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=2045cf1c-fb5a-4510-bf20-952306dbc126 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52921/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:05.751 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-03-11T22:46:05.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-03-11T22:46:05.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:05.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:05.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:05.762 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:05.759+0000 7fe296380ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:05.764 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:05.761+0000 7fe296380ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:05.765 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:05.762+0000 7fe296380ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:06.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-03-11T22:46:06.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:06.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:46:06.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:06.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:06.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:06.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:06.158 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:06.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:06.159 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:46:06.281 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:06.278+0000 7fe296380ec0 -1 Falling back to public interface 2020-03-11T22:46:06.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:07.182 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:07.178+0000 7fe296380ec0 -1 osd.3 26 log_to_monitors {default=true} 2020-03-11T22:46:07.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:07.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:07.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:07.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:07.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:07.525 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:46:07.895 INFO:tasks.workunit.client.0.smithi016.stdout:osd.3 up in weight 1 up_from 34 up_thru 0 down_at 29 last_clean_interval [20,26) [v2:127.0.0.1:6826/55342,v1:127.0.0.1:6827/55342] [v2:127.0.0.1:6828/55342,v1:127.0.0.1:6829/55342] exists,up e06624c8-444b-4300-9b2a-6571f16cacac 2020-03-11T22:46:07.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:07.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:07.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:07.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-03-11T22:46:07.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:07.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:07.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:07.897 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:07.898 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:07.898 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:07.898 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:07.898 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:07.898 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:07.898 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:07.988 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:07.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:07.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:07.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:07.989 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:08.373 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:08.373 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:46:08.373 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:46:08.373 INFO:tasks.workunit.client.0.smithi016.stderr:3' 2020-03-11T22:46:08.373 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:08.374 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:08.374 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:08.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=120259084292 2020-03-11T22:46:08.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084292' 2020-03-11T22:46:08.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:08.502 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:08.698 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=133143986179 2020-03-11T22:46:08.699 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084292 1-133143986179' 2020-03-11T22:46:08.699 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:08.699 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:08.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=137438953475 2020-03-11T22:46:08.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084292 1-133143986179 2-137438953475' 2020-03-11T22:46:08.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:08.826 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:46:08.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:46:08.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084292 1-133143986179 2-137438953475 3-146028888067' 2020-03-11T22:46:08.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:08.953 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-120259084292 2020-03-11T22:46:08.953 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:08.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:08.955 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-120259084292 2020-03-11T22:46:08.955 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:08.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=120259084292 2020-03-11T22:46:08.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 120259084292' 2020-03-11T22:46:08.956 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 120259084292 2020-03-11T22:46:08.957 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:09.322 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 120259084292 -lt 120259084292 2020-03-11T22:46:09.322 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:09.323 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-133143986179 2020-03-11T22:46:09.323 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:09.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:09.325 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-133143986179 2020-03-11T22:46:09.325 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:09.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=133143986179 2020-03-11T22:46:09.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2020-03-11T22:46:09.327 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 133143986179 2020-03-11T22:46:09.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:09.697 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 133143986179 2020-03-11T22:46:09.697 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:10.698 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:10.699 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:11.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 133143986180 -lt 133143986179 2020-03-11T22:46:11.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:11.072 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-137438953475 2020-03-11T22:46:11.072 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:11.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:46:11.074 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-137438953475 2020-03-11T22:46:11.074 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:11.075 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=137438953475 2020-03-11T22:46:11.075 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 137438953475' 2020-03-11T22:46:11.075 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 137438953475 2020-03-11T22:46:11.075 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:46:11.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 137438953476 -lt 137438953475 2020-03-11T22:46:11.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:11.438 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-146028888067 2020-03-11T22:46:11.438 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:11.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:46:11.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-146028888067 2020-03-11T22:46:11.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:11.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:46:11.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 146028888067' 2020-03-11T22:46:11.442 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 146028888067 2020-03-11T22:46:11.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:46:11.799 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888067 -lt 146028888067 2020-03-11T22:46:11.800 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:11.800 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:11.800 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:12.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:46:12.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:12.310 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:12.310 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:12.310 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:12.310 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:12.311 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:12.311 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:12.659 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:46:12.660 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:12.660 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:12.660 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:13.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:46:13.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:13.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:13.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-03-11T22:46:13.124 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:46:13.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:13.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:46:13.125 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:13.126 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:13.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:13.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:13.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:13.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:13.234 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:13.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:13.235 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:13.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:13.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:13.236 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:13.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:13.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:13.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:13.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:13.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:46:13.245 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:13.246 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.246 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:46:13.246 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52921 2020-03-11T22:46:13.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:13.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:13.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:46:13.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/ec-error-rollforward 0 2020-03-11T22:46:13.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:46:13.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:46:13.248 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:46:13.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:13.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:13.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:13.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:13.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:13.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:13.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:13.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:13.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:13.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:13.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:13.251 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:13.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:13.253 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:13.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:46:13.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:46:13.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:13.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52921 2020-03-11T22:46:13.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52921 2020-03-11T22:46:13.257 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:13.257 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:13.257 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:46:13.258 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:46:13.258 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:46:13.333 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-03-11T22:46:13.333 INFO:teuthology.orchestra.run.smithi016:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-03-11T22:46:13.388 INFO:tasks.workunit.client.0.smithi016.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:46:13.400 INFO:tasks.workunit.client.0.smithi016.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:46:13.401 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:13.401 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:13.401 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:46:13.401 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:46:13.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:46:13.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:46:13.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:46:13.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:46:13.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-prio 2020-03-11T22:46:13.403 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-03-11T22:46:13.403 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-03-11T22:46:13.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-03-11T22:46:13.406 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-03-11T22:46:13.406 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-03-11T22:46:13.407 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-03-11T22:46:13.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none ' 2020-03-11T22:46:13.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:46:13.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:13.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-03-11T22:46:13.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-03-11T22:46:13.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-03-11T22:46:13.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-03-11T22:46:13.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-03-11T22:46:13.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-03-11T22:46:13.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-03-11T22:46:13.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-03-11T22:46:13.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-03-11T22:46:13.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-03-11T22:46:13.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-03-11T22:46:13.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:46:13.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-03-11T22:46:13.410 INFO:tasks.workunit.client.0.smithi016.stderr:TEST_backfill_priority' 2020-03-11T22:46:13.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:46:13.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:46:13.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:46:13.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:46:13.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:46:13.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:13.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:46:13.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:13.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:13.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:13.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:13.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:13.415 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:13.416 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:13.416 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:13.416 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:13.416 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:13.416 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:13.417 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:13.417 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:46:13.418 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:13.418 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.418 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:46:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56083 2020-03-11T22:46:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:46:13.419 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:46:13.419 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.420 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:46:13.420 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56083 2020-03-11T22:46:13.420 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:46:13.420 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:46:13.420 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:46:13.420 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:46:13.420 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:46:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-03-11T22:46:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-03-11T22:46:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-03-11T22:46:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-03-11T22:46:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:46:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:46:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:46:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:46:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:46:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:46:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:46:13.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:46:13.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:13.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:13.737 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:13.737 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.737 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:46:13.738 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:46:13.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:46:13.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:46:13.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:46:13.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:13.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:13.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:46:13.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:46:13.924 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:13.924 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:13.924 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:13.924 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:13.924 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.924 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:46:13.924 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-mon.a.asok 2020-03-11T22:46:13.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:13.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56083/ceph-mon.a.asok config get fsid 2020-03-11T22:46:14.015 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:46:14.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:14.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:14.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:46:14.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:46:14.016 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:14.016 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:14.016 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:14.017 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:14.017 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:14.017 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:46:14.017 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-mon.a.asok 2020-03-11T22:46:14.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:14.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56083/ceph-mon.a.asok config get mon_host 2020-03-11T22:46:14.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:46:14.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:46:14.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:46:14.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:46:14.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:46:14.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:46:14.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:46:15.032 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:46:15.032 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:15.032 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:15.032 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:15.033 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:15.033 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:46:15.033 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:46:15.033 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:15.033 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:15.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-03-11T22:46:15.057 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-03-11T22:46:15.059 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-03-11T22:46:15.062 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:46:15.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:46:15.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:46:15.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:15.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:46:15.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:15.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:46:15.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:15.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:15.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:15.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:15.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:46:15.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:46:15.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:15.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:15.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:46:15.065 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:15.065 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:15.065 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:15.065 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:15.066 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:15.066 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:46:15.066 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:46:15.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:46:15.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:15.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:15.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:15.067 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:46:15.067 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:46:15.067 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:15.067 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:15.067 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:15.067 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:15.067 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:15.068 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:46:15.068 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:15.068 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=aa58bb65-fcea-43f5-a0b8-31e104b4fc1e 2020-03-11T22:46:15.069 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 aa58bb65-fcea-43f5-a0b8-31e104b4fc1e' 2020-03-11T22:46:15.069 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:15.069 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 aa58bb65-fcea-43f5-a0b8-31e104b4fc1e 2020-03-11T22:46:15.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA3amle2CPvBBAAYrv9IXfPm+ACX7NNJWSmSw== 2020-03-11T22:46:15.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA3amle2CPvBBAAYrv9IXfPm+ACX7NNJWSmSw=="}' 2020-03-11T22:46:15.086 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new aa58bb65-fcea-43f5-a0b8-31e104b4fc1e -i td/osd-backfill-prio/0/new.json 2020-03-11T22:46:15.405 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:46:15.414 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:46:15.415 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA3amle2CPvBBAAYrv9IXfPm+ACX7NNJWSmSw== --osd-uuid aa58bb65-fcea-43f5-a0b8-31e104b4fc1e 2020-03-11T22:46:15.442 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:15.439+0000 7f0570b1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:15.444 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:15.441+0000 7f0570b1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:15.446 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:15.442+0000 7f0570b1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:15.446 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:15.443+0000 7f0570b1cec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:15.446 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:15.443+0000 7f0570b1cec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:15.447 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:15.443+0000 7f0570b1cec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:15.544 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:15.541+0000 7f0570b1cec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:46:18.638 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:46:18.638 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:18.639 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:46:18.639 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:18.639 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:46:19.121 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:46:19.122 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:46:19.122 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:19.122 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:19.122 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:19.122 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:19.146 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:19.143+0000 7f7064f83ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:19.147 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:19.145+0000 7f7064f83ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:19.149 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:19.146+0000 7f7064f83ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:19.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:46:19.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:19.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:19.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:19.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:19.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:19.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:19.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:19.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:19.491 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:46:19.664 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:19.662+0000 7f7064f83ec0 -1 Falling back to public interface 2020-03-11T22:46:19.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:20.423 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:20.420+0000 7f7064f83ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:46:20.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:20.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:20.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:20.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:20.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:20.888 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:46:21.323 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56515,v1:127.0.0.1:6803/56515] [v2:127.0.0.1:6804/56515,v1:127.0.0.1:6805/56515] exists,up aa58bb65-fcea-43f5-a0b8-31e104b4fc1e 2020-03-11T22:46:21.324 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 894b7884-8299-4931-b433-e2c030674863 2020-03-11T22:46:21.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:21.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:21.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:21.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:46:21.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:46:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:46:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:46:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:46:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:46:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:46:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:46:21.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:21.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:21.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:46:21.328 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:46:21.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:46:21.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:21.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:21.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:21.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:46:21.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:46:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:46:21.329 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:21.330 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=894b7884-8299-4931-b433-e2c030674863 2020-03-11T22:46:21.330 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 894b7884-8299-4931-b433-e2c030674863' 2020-03-11T22:46:21.330 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:21.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA9amlec74WFBAA0pfPFXA8/eI3EN5EG5fNmg== 2020-03-11T22:46:21.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA9amlec74WFBAA0pfPFXA8/eI3EN5EG5fNmg=="}' 2020-03-11T22:46:21.340 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 894b7884-8299-4931-b433-e2c030674863 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:46:21.809 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:46:21.819 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:46:21.820 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA9amlec74WFBAA0pfPFXA8/eI3EN5EG5fNmg== --osd-uuid 894b7884-8299-4931-b433-e2c030674863 2020-03-11T22:46:21.846 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:21.844+0000 7ff95d573ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.859 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:21.857+0000 7ff95d573ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.861 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:21.858+0000 7ff95d573ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.861 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:21.859+0000 7ff95d573ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:21.862 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:21.859+0000 7ff95d573ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:21.862 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:21.859+0000 7ff95d573ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:21.917 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:21.915+0000 7ff95d573ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:46:24.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:46:24.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:24.986 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:46:24.986 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:46:24.986 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:25.481 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:46:25.481 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:25.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:25.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:25.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:25.482 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:46:25.506 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:25.504+0000 7f5edcbe5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.507 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:25.505+0000 7f5edcbe5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.509 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:25.506+0000 7f5edcbe5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:46:25.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:25.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:25.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:25.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:25.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:25.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:25.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:25.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:25.848 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:46:26.026 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:26.023+0000 7f5edcbe5ec0 -1 Falling back to public interface 2020-03-11T22:46:26.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:26.851 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:46:26.848+0000 7f5edcbe5ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:46:27.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:27.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:27.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:27.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:27.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:27.220 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:46:27.603 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/56870,v1:127.0.0.1:6811/56870] [v2:127.0.0.1:6812/56870,v1:127.0.0.1:6813/56870] exists,up 894b7884-8299-4931-b433-e2c030674863 2020-03-11T22:46:27.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:27.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:27.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:27.604 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:27.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:27.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-03-11T22:46:27.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:46:28.229 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test1' already exists 2020-03-11T22:46:28.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:29.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:46:30.122 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 size to 2 2020-03-11T22:46:30.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:30.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-03-11T22:46:30.130 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:46:30.742 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test2' already exists 2020-03-11T22:46:30.750 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:31.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:46:32.584 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 2 size to 2 2020-03-11T22:46:32.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:32.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-03-11T22:46:32.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:46:33.140 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test3' already exists 2020-03-11T22:46:33.148 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:34.149 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-03-11T22:46:34.838 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 3 size to 2 2020-03-11T22:46:34.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:46:39.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:46:39.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:39.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:39.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:39.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:39.849 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:39.850 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:39.850 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:39.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:39.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:39.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:39.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:39.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:39.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:39.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:39.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:40.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:40.326 INFO:tasks.workunit.client.0.smithi016.stderr:1' 2020-03-11T22:46:40.327 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:40.327 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:40.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:40.459 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:46:40.459 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:46:40.459 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:40.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:40.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:46:40.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:46:40.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:40.593 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:46:40.593 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:40.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:40.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:46:40.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:40.596 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:46:40.596 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:46:40.596 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:46:40.597 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:40.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:46:40.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:40.966 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:46:40.966 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:40.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:40.968 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:46:40.968 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:40.969 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:46:40.969 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:46:40.969 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:46:40.969 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:41.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672964 -lt 42949672966 2020-03-11T22:46:41.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:42.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:42.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:42.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:46:42.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:42.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:42.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:43.247 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:46:43.247 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:43.248 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:43.248 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:43.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:43.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:43.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:43.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:43.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:46:43.623 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:43.623 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:43.623 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:44.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:46:44.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:44.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:44.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:46:44.442 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:44.443 INFO:tasks.workunit.client.0.smithi016.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:35.564059+0000 0'0 27:17 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:46:32.918868+0000 0'0 2020-03-11T22:46:32.918868+0000 0 2020-03-11T22:46:44.443 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:30.056536+0000 0'0 26:27 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:46:27.995660+0000 0'0 2020-03-11T22:46:27.995660+0000 0 2020-03-11T22:46:44.443 INFO:tasks.workunit.client.0.smithi016.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:32.896784+0000 0'0 26:22 [0,1] 0 [0,1] 0 0'0 2020-03-11T22:46:30.483485+0000 0'0 2020-03-11T22:46:30.483485+0000 0 2020-03-11T22:46:44.443 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:46:44.443 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:44.444 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:46:44.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-03-11T22:46:44.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-03-11T22:46:44.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-03-11T22:46:44.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-03-11T22:46:44.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-03-11T22:46:44.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-03-11T22:46:44.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-03-11T22:46:44.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-03-11T22:46:44.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-03-11T22:46:44.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-03-11T22:46:44.455 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:44.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:44.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-03-11T22:46:44.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:46:44.810 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:46:44.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-03-11T22:46:44.811 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:46:44.813 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-03-11T22:46:44.813 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-03-11T22:46:44.813 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-03-11T22:46:44.813 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-03-11T22:46:44.813 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-03-11T22:46:44.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-03-11T22:46:44.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-03-11T22:46:44.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:44.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-03-11T22:46:44.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:46:45.260 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:46:45.261 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-03-11T22:46:45.262 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:46:45.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-03-11T22:46:45.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-03-11T22:46:45.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-03-11T22:46:45.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-03-11T22:46:45.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-03-11T22:46:45.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-03-11T22:46:45.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-03-11T22:46:45.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-03-11T22:46:45.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-03-11T22:46:45.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:46:45.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-03-11T22:46:45.266 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:45.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:45.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-03-11T22:46:45.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:45.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-03-11T22:46:45.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:45.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-03-11T22:46:45.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-03-11T22:46:45.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:46:45.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:46:45.799 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test3' does not exist 2020-03-11T22:46:45.807 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-03-11T22:46:45.807 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-03-11T22:46:45.808 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-03-11T22:46:45.808 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-03-11T22:46:45.810 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-03-11T22:46:45.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-03-11T22:46:45.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-03-11T22:46:46.522 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 size to 1 2020-03-11T22:46:46.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-03-11T22:46:47.198 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 recovery_priority to 1 2020-03-11T22:46:47.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-03-11T22:46:48.000 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 2 size to 1 2020-03-11T22:46:48.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-03-11T22:46:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 2 recovery_priority to 2 2020-03-11T22:46:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:46:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:48.779 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:48.779 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:48.779 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:48.780 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:48.874 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:49.247 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:49.247 INFO:tasks.workunit.client.0.smithi016.stderr:1' 2020-03-11T22:46:49.247 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:49.247 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:49.248 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:49.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:46:49.393 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:46:49.393 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:49.393 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:49.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672970 2020-03-11T22:46:49.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672970' 2020-03-11T22:46:49.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:49.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:46:49.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:49.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:49.528 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:46:49.528 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:49.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:46:49.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:46:49.530 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:46:49.530 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:49.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:46:49.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:50.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:50.896 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:51.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:46:51.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:51.273 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672970 2020-03-11T22:46:51.274 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:51.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:51.276 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672970 2020-03-11T22:46:51.276 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:51.277 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672970 2020-03-11T22:46:51.277 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-03-11T22:46:51.277 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672970 2020-03-11T22:46:51.277 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:51.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672970 -lt 42949672970 2020-03-11T22:46:51.680 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:51.681 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:51.681 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:52.167 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:46:52.167 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:52.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:52.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:52.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:52.169 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:52.169 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:52.169 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:52.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:46:52.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:52.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:52.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:53.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:46:53.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:53.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:53.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:46:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:10+0 records in 2020-03-11T22:46:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:10+0 records out 2020-03-11T22:46:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0461725 s, 227 MB/s 2020-03-11T22:46:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-03-11T22:46:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:46:53.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:46:53.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:53.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:46:53.301 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:53.301 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:46:53.571 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:53.572 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:46:53.801 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:53.801 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:46:54.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:54.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:46:54.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:54.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:46:54.752 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:54.752 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:46:55.043 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:46:55.368 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.368 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:46:55.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:46:55.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.478 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.805 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.805 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.340 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.586 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.586 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.829 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.137 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.137 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.395 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.395 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.538 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.538 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.821 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.088 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.088 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.389 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.389 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.677 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.214 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.214 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.528 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.148 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:47:05.048 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.048 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:47:05.507 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.508 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:47:05.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:47:06.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:47:06.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:47:06.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:47:06.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:47:07.132 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.133 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:47:07.425 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-03-11T22:47:07.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-03-11T22:47:07.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:47:07.427 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:47:07.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:47:07.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:47:07.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:47:08.148 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.148 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:47:08.447 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.447 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:47:08.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.041 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.306 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.598 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.290 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.416 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.416 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.275 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.275 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.377 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.703 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.703 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.509 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.509 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.972 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.972 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.843 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.843 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.102 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.102 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.813 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:47:19.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:19.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:47:19.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:19.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:47:19.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:19.686 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:47:19.944 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:19.945 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:47:20.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:20.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:47:20.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:20.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:47:20.807 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:20.807 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:47:21.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:21.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:47:21.425 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:21.425 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:47:21.679 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-03-11T22:47:21.680 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-03-11T22:47:21.682 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-03-11T22:47:21.682 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:47:21.682 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:47:21.682 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:47:21.683 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:47:21.683 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:47:21.683 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:47:21.683 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:47:22.062 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:47:22.062 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:47:22.062 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:47:22.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-03-11T22:47:22.430 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:47:22.774 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:22.774 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:46:46.784163+0000 37'150 37:193 [1] 1 [1] 1 0'0 2020-03-11T22:46:27.995660+0000 0'0 2020-03-11T22:46:27.995660+0000 0 2020-03-11T22:47:22.774 INFO:tasks.workunit.client.0.smithi016.stdout:2.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:46:48.012352+0000 37'150 37:186 [0] 0 [0] 0 0'0 2020-03-11T22:46:30.483485+0000 0'0 2020-03-11T22:46:30.483485+0000 0 2020-03-11T22:47:22.775 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:47:22.775 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:22.776 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:47:22.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-03-11T22:47:22.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:47:23.378 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 size to 2 2020-03-11T22:47:23.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:47:24.086 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 2 size to 2 2020-03-11T22:47:24.094 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:47:29.097 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-03-11T22:47:29.097 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:47:29.097 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:47:29.098 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:29.098 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:29.098 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:47:29.098 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:47:29.098 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:47:29.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:47:29.243 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 2020-03-11T22:47:29.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-03-11T22:47:29.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:29.244 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:47:29.244 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:47:29.244 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:47:29.244 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:47:29.244 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:47:29.245 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:47:29.245 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:47:29.245 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:47:29.245 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 152, 2020-03-11T22:47:29.245 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:47:29.245 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:47:29.245 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:47:29.246 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:47:29.246 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:47:29.246 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:47:29.246 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:47:29.246 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:47:29.246 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:47:29.246 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:47:29.246 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "2.0", 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 153, 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:47:29.248 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-03-11T22:47:29.248 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:47:29.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:47:29.249 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:29.249 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:29.249 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:47:29.249 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.0.asok 2020-03-11T22:47:29.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:47:29.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.0.asok dump_recovery_reservations 2020-03-11T22:47:29.347 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 2020-03-11T22:47:29.348 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-03-11T22:47:29.348 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:29.348 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:47:29.348 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "2.0", 2020-03-11T22:47:29.350 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 153, 2020-03-11T22:47:29.350 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:47:29.350 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:47:29.350 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:47:29.350 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:47:29.350 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:47:29.350 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:47:29.350 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:47:29.351 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:47:29.351 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:47:29.351 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:47:29.351 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:47:29.351 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 152, 2020-03-11T22:47:29.351 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:47:29.351 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:47:29.352 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:47:29.352 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:47:29.352 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:47:29.353 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:29.353 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:47:29.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:47:29.353 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:47:29.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:47:29.354 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:29.354 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:47:29.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:47:29.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:47:29.356 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:29.356 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:47:29.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:47:29.358 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:47:29.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:47:29.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:29.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:47:29.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:47:29.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:47:29.363 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:29.363 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:47:29.365 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:47:29.365 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:47:29.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:47:29.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:29.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:47:29.368 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:47:29.369 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:47:29.370 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:29.370 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:47:29.372 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:47:29.372 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:47:29.373 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:47:29.373 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:29.373 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:47:29.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:47:29.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:47:29.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:47:29.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:29.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:29.377 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:29.377 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:29.378 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:29.378 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:29.378 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:29.378 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:29.378 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:29.378 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:29.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:29.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:29.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:29.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:29.471 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:29.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:29.932 INFO:tasks.workunit.client.0.smithi016.stderr:1' 2020-03-11T22:47:29.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:29.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:29.933 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:30.069 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836500 2020-03-11T22:47:30.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836500' 2020-03-11T22:47:30.070 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:30.070 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:30.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672980 2020-03-11T22:47:30.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836500 1-42949672980' 2020-03-11T22:47:30.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:30.202 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836500 2020-03-11T22:47:30.202 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:30.203 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:30.204 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836500 2020-03-11T22:47:30.204 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:30.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836500 2020-03-11T22:47:30.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2020-03-11T22:47:30.205 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836500 2020-03-11T22:47:30.206 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:30.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836498 -lt 21474836500 2020-03-11T22:47:30.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:31.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:31.583 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:31.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836500 -lt 21474836500 2020-03-11T22:47:31.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:31.960 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672980 2020-03-11T22:47:31.961 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:31.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:31.963 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672980 2020-03-11T22:47:31.963 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:31.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672980 2020-03-11T22:47:31.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2020-03-11T22:47:31.964 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672980 2020-03-11T22:47:31.964 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:32.342 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672980 -lt 42949672980 2020-03-11T22:47:32.343 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:32.343 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:32.343 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:33.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:47:33.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:33.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:33.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:33.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:33.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:33.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:33.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:33.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:33.386 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:33.387 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:33.387 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:33.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:33.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:47:33.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:47:33.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:47:33.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:33.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:33.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:33.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:33.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:33.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:33.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:33.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:33.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:33.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:34.390 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:34.391 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:34.391 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:34.391 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:34.926 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:34.926 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:34.926 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:34.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:34.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:34.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:34.927 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:34.927 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:34.928 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:35.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26208915 2020-03-11T22:47:35.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26208915 '!=' null 2020-03-11T22:47:35.425 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:35.425 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:35.425 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:35.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:35.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:35.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:35.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:35.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:35.528 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:35.528 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:35.528 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:35.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:35.883 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:35.884 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:35.884 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:36.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:36.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:36.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:36.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:36.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:36.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:36.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:36.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:36.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:36.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26208915 2020-03-11T22:47:36.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26208915 '!=' null 2020-03-11T22:47:36.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:36.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:36.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:36.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:36.947 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:36.948 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:36.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:36.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:36.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:36.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:36.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:37.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:37.317 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:37.317 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:37.317 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:37.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:37.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:37.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:37.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:37.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:37.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:37.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:37.985 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:37.985 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:38.587 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22237201 2020-03-11T22:47:38.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22237201 '!=' null 2020-03-11T22:47:38.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:38.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:38.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:38.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:38.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:38.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:38.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:38.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:38.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:38.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:38.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:39.051 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:39.052 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:39.052 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:39.052 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:39.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:39.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:39.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:39.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:39.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:39.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:39.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:39.530 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:39.530 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:40.075 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34945743 2020-03-11T22:47:40.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34945743 '!=' null 2020-03-11T22:47:40.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:40.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:40.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:40.176 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:40.176 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:40.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:40.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:40.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:40.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:40.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:40.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:40.740 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:40.741 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:40.741 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:40.741 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:41.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:41.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:41.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:41.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:41.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:41.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:41.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:41.269 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:41.269 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:41.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27956993 2020-03-11T22:47:41.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27956993 '!=' null 2020-03-11T22:47:41.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:41.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:41.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:41.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:41.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:41.848 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:41.848 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:41.848 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:41.848 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:41.848 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:41.849 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:42.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:42.213 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:42.213 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:42.214 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:42.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:42.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:42.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:42.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:42.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:42.690 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:42.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:42.691 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:42.691 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:43.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27083686 2020-03-11T22:47:43.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27083686 '!=' null 2020-03-11T22:47:43.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:43.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:43.292 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:43.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:43.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:43.394 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:43.394 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:43.394 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:43.394 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:43.394 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:43.394 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:43.752 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:43.753 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:43.753 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:43.754 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:44.258 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:44.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:44.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:44.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:44.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:44.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:44.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:44.260 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:44.260 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:44.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27083686 2020-03-11T22:47:44.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27083686 '!=' null 2020-03-11T22:47:44.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:44.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:44.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:44.997 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:44.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:44.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:44.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:44.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:44.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:44.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:44.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:45.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:45.361 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:45.362 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:45.362 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:45.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:45.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:45.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:45.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:45.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:45.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:45.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:45.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:45.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:46.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31452306 2020-03-11T22:47:46.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31452306 '!=' null 2020-03-11T22:47:46.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:46.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:46.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:46.573 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:46.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:46.575 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:46.575 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:46.575 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:46.575 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:46.575 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:46.575 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:47.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:47.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:47.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:47.031 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:47.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:47.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:47.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:47.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:47.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:47.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:47.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:47.525 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:47.525 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:48.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27083998 2020-03-11T22:47:48.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27083998 '!=' null 2020-03-11T22:47:48.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:48.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:48.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:48.184 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:48.185 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:48.186 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:48.186 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:48.186 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:48.186 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:48.186 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:48.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:48.572 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:48.573 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:48.573 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:49.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:49.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:49.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:49.107 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:49.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:49.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:49.108 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:49.108 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:49.108 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:49.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41062672 2020-03-11T22:47:49.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41062672 '!=' null 2020-03-11T22:47:49.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:49.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:49.896 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:49.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:49.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:49.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:49.998 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:49.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:49.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:49.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:49.999 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:50.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:50.372 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:50.373 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:50.373 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:50.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:50.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:50.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:50.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:50.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:50.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:50.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:50.847 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:50.847 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:51.342 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957485 2020-03-11T22:47:51.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957485 '!=' null 2020-03-11T22:47:51.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:51.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:51.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:51.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:51.443 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:51.444 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:51.444 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:51.445 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:51.445 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:51.445 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:51.445 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:51.804 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:51.805 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:51.805 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:51.805 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:52.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:52.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:52.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:52.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:52.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:52.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:52.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:52.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:52.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:53.043 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946990 2020-03-11T22:47:53.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946990 '!=' null 2020-03-11T22:47:53.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:53.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:53.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:53.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:53.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:53.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:53.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:53.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:53.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:53.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:53.147 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:53.512 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:53.513 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:53.513 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:53.513 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:54.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:54.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:54.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:54.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:54.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:54.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:54.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:54.006 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:54.006 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:54.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946990 2020-03-11T22:47:54.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946990 '!=' null 2020-03-11T22:47:54.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:54.907 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:54.907 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:55.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:55.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:55.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:55.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:55.008 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:55.009 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:55.009 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:55.009 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:55.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:55.372 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:55.373 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:55.373 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:55.900 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:55.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:55.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:55.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:55.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:55.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:55.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:55.902 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:55.902 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:56.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946972 2020-03-11T22:47:56.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946972 '!=' null 2020-03-11T22:47:56.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:56.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:56.382 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:56.481 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:56.481 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:56.482 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:56.482 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:56.482 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:56.483 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:56.483 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:56.483 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:56.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:56.848 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:56.848 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:56.849 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:57.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:57.328 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:57.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:57.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:57.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:57.329 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:57.330 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:57.330 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:57.330 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:57.802 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20968269 2020-03-11T22:47:57.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20968269 '!=' null 2020-03-11T22:47:57.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:57.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:57.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:57.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:57.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:57.904 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:57.904 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:57.904 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:57.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:57.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:57.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:58.258 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:58.258 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:58.259 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:58.259 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:58.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:58.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:58.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:58.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:58.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:58.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:58.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:58.734 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:58.734 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:59.242 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31452329 2020-03-11T22:47:59.242 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31452329 '!=' null 2020-03-11T22:47:59.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:59.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:59.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:59.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:59.343 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:59.344 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:59.344 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:59.345 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:59.345 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:59.345 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:59.345 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:59.700 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:00.178 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:48:00.178 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:48:00.178 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:48:00.179 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:48:00.179 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:00.179 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:00.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:00.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:00.280 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:00.281 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:00.281 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:00.281 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:00.281 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:00.282 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:00.635 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:48:00.636 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:00.636 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:00.636 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:01.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:48:01.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:48:01.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:01.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:01.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:01.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:01.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:01.115 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:01.115 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:01.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25796209 2020-03-11T22:48:01.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25796209 '!=' null 2020-03-11T22:48:01.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:01.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:01.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:01.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:01.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:01.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:01.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:01.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:01.690 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:01.691 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:01.691 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:02.041 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:48:02.043 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:02.044 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:02.044 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:02.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:03.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19362495 2020-03-11T22:48:03.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19362495 '!=' null 2020-03-11T22:48:03.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:03.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:03.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:03.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:03.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:03.160 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:03.160 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:03.161 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:03.161 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:03.161 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:03.161 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:03.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:48:03.515 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:03.515 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:03.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:03.994 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:48:03.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:48:03.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:48:03.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:48:03.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-03-11T22:48:03.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-03-11T22:48:03.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:48:03.996 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:48:03.996 INFO:tasks.workunit.client.0.smithi016.stdout:TEST PASSED 2020-03-11T22:48:04.495 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test1' does not exist 2020-03-11T22:48:04.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-03-11T22:48:04.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:48:04.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:48:05.133 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test2' does not exist 2020-03-11T22:48:05.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:48:05.142 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:05.142 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:05.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:05.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:05.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:05.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:05.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-03-11T22:48:05.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:48:05.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:48:05.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:48:05.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:48:05.464 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:05.464 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:05.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:05.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:05.465 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:05.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:05.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:05.468 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:05.468 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:05.469 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:05.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:05.470 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:05.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:05.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:05.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:05.472 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:05.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:05.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:48:05.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:48:05.486 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:05.486 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:05.486 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:05.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56083 2020-03-11T22:48:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:48:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:48:05.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:48:05.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:48:05.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:48:05.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:48:05.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:48:05.489 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:05.489 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:05.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:05.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:05.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:05.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:05.491 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:05.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:05.491 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:05.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:05.492 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:05.492 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:05.492 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:05.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:05.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:05.494 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:05.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:05.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:48:05.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:48:05.496 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:05.496 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:05.496 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:05.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56083 2020-03-11T22:48:05.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:05.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:05.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:48:05.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:48:05.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:05.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:05.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56083 2020-03-11T22:48:05.500 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:48:05.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:48:05.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:48:05.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:48:05.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-03-11T22:48:05.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-03-11T22:48:05.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-03-11T22:48:05.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-03-11T22:48:05.501 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-03-11T22:48:05.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-03-11T22:48:05.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-03-11T22:48:05.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:48:05.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:48:05.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:48:05.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:48:05.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:48:05.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:48:05.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:48:05.839 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:48:05.839 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:05.839 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:05.839 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:05.839 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:05.840 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:05.840 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:05.840 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:48:06.048 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:48:06.049 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:48:06.049 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:48:06.049 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:48:06.049 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:48:06.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:48:06.051 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:48:06.051 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:48:06.051 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:48:06.051 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:06.051 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:06.052 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:06.052 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-mon.a.asok 2020-03-11T22:48:06.052 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:48:06.052 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56083/ceph-mon.a.asok config get fsid 2020-03-11T22:48:06.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:48:06.147 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:48:06.147 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:48:06.147 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:48:06.147 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:48:06.147 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:48:06.147 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:48:06.148 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:48:06.148 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:06.148 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:06.148 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:06.148 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-mon.a.asok 2020-03-11T22:48:06.148 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:48:06.148 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56083/ceph-mon.a.asok config get mon_host 2020-03-11T22:48:06.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:48:06.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:48:06.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:48:06.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:48:06.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:48:06.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:48:06.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:48:07.162 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:48:07.162 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:07.162 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:07.162 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:07.162 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:07.162 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:07.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:07.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:48:07.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:48:07.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-03-11T22:48:07.187 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-03-11T22:48:07.188 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-03-11T22:48:07.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:07.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:48:07.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:07.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:07.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:48:07.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:07.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:48:07.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:07.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:07.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:07.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:07.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:48:07.193 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:48:07.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:07.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:07.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:07.194 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:07.194 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:07.194 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:07.194 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:07.195 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:07.195 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:07.195 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:07.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:07.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:07.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:07.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:07.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:07.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:07.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:07.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:07.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:07.196 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:07.197 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:07.197 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:48:07.198 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:07.199 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1c5e14c3-fa7c-4520-924f-9d5a830044dd 2020-03-11T22:48:07.199 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 1c5e14c3-fa7c-4520-924f-9d5a830044dd' 2020-03-11T22:48:07.199 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:07.199 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 1c5e14c3-fa7c-4520-924f-9d5a830044dd 2020-03-11T22:48:07.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCnamleyBL4DBAANjT0Pyo+pT8Dk6etq2Lcbg== 2020-03-11T22:48:07.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCnamleyBL4DBAANjT0Pyo+pT8Dk6etq2Lcbg=="}' 2020-03-11T22:48:07.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1c5e14c3-fa7c-4520-924f-9d5a830044dd -i td/osd-backfill-prio/0/new.json 2020-03-11T22:48:07.531 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:48:07.541 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:48:07.542 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCnamleyBL4DBAANjT0Pyo+pT8Dk6etq2Lcbg== --osd-uuid 1c5e14c3-fa7c-4520-924f-9d5a830044dd 2020-03-11T22:48:07.569 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:07.565+0000 7fab71a3aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:07.570 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:07.567+0000 7fab71a3aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:07.572 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:07.568+0000 7fab71a3aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:07.572 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:07.569+0000 7fab71a3aec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:07.572 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:07.569+0000 7fab71a3aec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:07.573 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:07.569+0000 7fab71a3aec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:07.647 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:07.644+0000 7fab71a3aec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:48:10.737 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:48:10.737 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:10.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:48:10.738 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:10.738 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:48:11.233 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:48:11.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:48:11.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:11.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:11.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:11.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:11.258 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:11.255+0000 7f198f8e3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:11.274 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:11.270+0000 7f198f8e3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:11.275 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:11.272+0000 7f198f8e3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:11.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:48:11.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:11.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:48:11.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:11.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:11.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:11.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:11.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:11.602 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:11.602 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:48:11.789 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:11.785+0000 7f198f8e3ec0 -1 Falling back to public interface 2020-03-11T22:48:12.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:12.530 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:12.527+0000 7f198f8e3ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:48:13.030 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:13.031 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:13.031 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:13.031 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:13.031 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:13.031 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:48:13.416 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/63064,v1:127.0.0.1:6803/63064] [v2:127.0.0.1:6804/63064,v1:127.0.0.1:6805/63064] exists,up 1c5e14c3-fa7c-4520-924f-9d5a830044dd 2020-03-11T22:48:13.417 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 2b528975-e4f2-480f-8b62-91e73823598c 2020-03-11T22:48:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:13.417 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:48:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:48:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:48:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:13.418 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:13.420 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:13.420 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:13.420 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:13.420 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:13.420 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:13.420 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:13.420 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:13.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:48:13.422 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:13.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2b528975-e4f2-480f-8b62-91e73823598c 2020-03-11T22:48:13.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 2b528975-e4f2-480f-8b62-91e73823598c' 2020-03-11T22:48:13.423 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:13.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCtamle7sqvGRAAXXz6m4vAnSqgEUfY4qb2Lg== 2020-03-11T22:48:13.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCtamle7sqvGRAAXXz6m4vAnSqgEUfY4qb2Lg=="}' 2020-03-11T22:48:13.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2b528975-e4f2-480f-8b62-91e73823598c -i td/osd-backfill-prio/1/new.json 2020-03-11T22:48:13.981 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:48:13.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:48:13.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCtamle7sqvGRAAXXz6m4vAnSqgEUfY4qb2Lg== --osd-uuid 2b528975-e4f2-480f-8b62-91e73823598c 2020-03-11T22:48:14.019 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:14.016+0000 7f029b9a6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:14.038 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:14.035+0000 7f029b9a6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:14.039 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:14.036+0000 7f029b9a6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:14.039 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:14.037+0000 7f029b9a6ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:14.039 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:14.037+0000 7f029b9a6ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:14.040 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:14.037+0000 7f029b9a6ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:14.097 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:14.093+0000 7f029b9a6ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:48:17.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:48:17.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:17.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:48:17.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:17.423 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:48:17.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:48:17.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:17.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:17.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:17.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:17.915 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:48:17.939 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:17.936+0000 7ff13c18cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:17.940 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:17.938+0000 7ff13c18cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:17.942 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:17.940+0000 7ff13c18cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:18.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:48:18.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:18.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:48:18.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:18.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:18.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:18.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:18.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:18.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:18.367 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:48:18.463 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:18.460+0000 7ff13c18cec0 -1 Falling back to public interface 2020-03-11T22:48:18.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:19.187 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:19.184+0000 7ff13c18cec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:48:19.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:19.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:19.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:19.769 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:48:19.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:19.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:20.149 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63419,v1:127.0.0.1:6811/63419] [v2:127.0.0.1:6812/63419,v1:127.0.0.1:6813/63419] exists,up 2b528975-e4f2-480f-8b62-91e73823598c 2020-03-11T22:48:20.149 INFO:tasks.workunit.client.0.smithi016.stdout:add osd2 c3a29515-78ab-4e1c-86d5-01826842f807 2020-03-11T22:48:20.149 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:20.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:20.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:20.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:20.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-03-11T22:48:20.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:20.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:20.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:48:20.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:20.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-03-11T22:48:20.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:20.151 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:20.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:20.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:20.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-03-11T22:48:20.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-03-11T22:48:20.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:20.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:20.152 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:20.153 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:20.153 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:20.153 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:20.153 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:20.153 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:20.154 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:20.154 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:20.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:20.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:20.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:20.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:20.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:20.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:20.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:20.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:20.155 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:20.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:20.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:20.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-03-11T22:48:20.156 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:20.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=c3a29515-78ab-4e1c-86d5-01826842f807 2020-03-11T22:48:20.156 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 c3a29515-78ab-4e1c-86d5-01826842f807' 2020-03-11T22:48:20.156 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC0amleNuWvCRAAXUTO63/oa4T8AQkIgtCFKA== 2020-03-11T22:48:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC0amleNuWvCRAAXUTO63/oa4T8AQkIgtCFKA=="}' 2020-03-11T22:48:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new c3a29515-78ab-4e1c-86d5-01826842f807 -i td/osd-backfill-prio/2/new.json 2020-03-11T22:48:20.636 INFO:tasks.workunit.client.0.smithi016.stdout:2 2020-03-11T22:48:20.646 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-03-11T22:48:20.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC0amleNuWvCRAAXUTO63/oa4T8AQkIgtCFKA== --osd-uuid c3a29515-78ab-4e1c-86d5-01826842f807 2020-03-11T22:48:20.673 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:20.671+0000 7ff51992fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:20.819 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:20.817+0000 7ff51992fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:20.821 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:20.818+0000 7ff51992fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:20.821 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:20.819+0000 7ff51992fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:20.822 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:20.819+0000 7ff51992fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:20.822 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:20.819+0000 7ff51992fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:20.953 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:20.950+0000 7ff51992fec0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-03-11T22:48:24.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-03-11T22:48:24.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:24.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:48:24.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:24.047 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd2 key to auth repository 2020-03-11T22:48:24.549 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:48:24.550 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:48:24.550 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:24.550 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:24.550 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:24.550 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:24.575 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:24.571+0000 7f222ecffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:24.576 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:24.573+0000 7f222ecffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:24.577 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:24.575+0000 7f222ecffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:24.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:48:24.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:24.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:48:24.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:24.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:24.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:24.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:24.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:24.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:24.921 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:48:25.092 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:25.089+0000 7f222ecffec0 -1 Falling back to public interface 2020-03-11T22:48:25.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:25.836 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:25.833+0000 7f222ecffec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:48:26.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:26.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:26.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:26.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:26.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:26.287 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:48:26.772 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63774,v1:127.0.0.1:6819/63774] [v2:127.0.0.1:6820/63774,v1:127.0.0.1:6821/63774] exists,up c3a29515-78ab-4e1c-86d5-01826842f807 2020-03-11T22:48:26.773 INFO:tasks.workunit.client.0.smithi016.stdout:add osd3 78a28a77-654a-4e53-a906-d9a996f28e7b 2020-03-11T22:48:26.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:26.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:26.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:26.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-03-11T22:48:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:48:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-03-11T22:48:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:26.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:26.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:26.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:26.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-03-11T22:48:26.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-03-11T22:48:26.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:26.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:26.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:26.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:26.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:26.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:26.776 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:26.776 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:26.776 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:26.777 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:26.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:26.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:26.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:26.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:26.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:26.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:26.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:26.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:26.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:26.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:26.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:26.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-03-11T22:48:26.780 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:26.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=78a28a77-654a-4e53-a906-d9a996f28e7b 2020-03-11T22:48:26.780 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 78a28a77-654a-4e53-a906-d9a996f28e7b' 2020-03-11T22:48:26.780 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:26.790 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC6amlerCDkLhAAJBMJuKRsRe53s8E3fE1U/A== 2020-03-11T22:48:26.790 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC6amlerCDkLhAAJBMJuKRsRe53s8E3fE1U/A=="}' 2020-03-11T22:48:26.790 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 78a28a77-654a-4e53-a906-d9a996f28e7b -i td/osd-backfill-prio/3/new.json 2020-03-11T22:48:27.249 INFO:tasks.workunit.client.0.smithi016.stdout:3 2020-03-11T22:48:27.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-03-11T22:48:27.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC6amlerCDkLhAAJBMJuKRsRe53s8E3fE1U/A== --osd-uuid 78a28a77-654a-4e53-a906-d9a996f28e7b 2020-03-11T22:48:27.287 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:27.284+0000 7f554cdbcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:27.302 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:27.300+0000 7f554cdbcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:27.303 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:27.301+0000 7f554cdbcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:27.303 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:27.301+0000 7f554cdbcec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:27.304 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:27.301+0000 7f554cdbcec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:27.304 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:27.301+0000 7f554cdbcec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:27.352 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:27.350+0000 7f554cdbcec0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-03-11T22:48:30.544 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-03-11T22:48:30.544 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:30.546 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd3 key to auth repository 2020-03-11T22:48:30.546 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:48:30.546 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:31.036 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:48:31.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:31.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:31.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:31.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:31.037 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.3 2020-03-11T22:48:31.062 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:31.059+0000 7f8d97b32ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:31.064 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:31.061+0000 7f8d97b32ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:31.065 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:31.062+0000 7f8d97b32ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:31.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:48:31.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:31.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:48:31.423 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:31.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:31.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:31.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:31.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:31.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:48:31.424 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:48:31.585 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:31.582+0000 7f8d97b32ec0 -1 Falling back to public interface 2020-03-11T22:48:31.807 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:32.352 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:32.349+0000 7f8d97b32ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:48:32.808 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:32.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:32.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:32.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:32.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:48:32.809 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:48:33.200 INFO:tasks.workunit.client.0.smithi016.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/64129,v1:127.0.0.1:6827/64129] [v2:127.0.0.1:6828/64129,v1:127.0.0.1:6829/64129] exists,up 78a28a77-654a-4e53-a906-d9a996f28e7b 2020-03-11T22:48:33.200 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:33.200 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:33.200 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:33.200 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:33.200 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-03-11T22:48:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:48:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-03-11T22:48:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:33.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:33.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:33.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-03-11T22:48:33.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-03-11T22:48:33.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:33.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:33.202 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:33.203 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:33.203 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:33.203 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:33.203 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:33.203 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:33.203 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:48:33.203 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:33.203 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' 2020-03-11T22:48:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:33.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:33.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:33.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:33.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-03-11T22:48:33.205 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:33.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3b65ff47-d96d-4084-819c-6d6d58dbdd82 2020-03-11T22:48:33.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 3b65ff47-d96d-4084-819c-6d6d58dbdd82' 2020-03-11T22:48:33.206 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:33.206 INFO:tasks.workunit.client.0.smithi016.stdout:add osd4 3b65ff47-d96d-4084-819c-6d6d58dbdd82 2020-03-11T22:48:33.217 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDBamleYQ3FDBAAZ20T6KZqz6uf3QE6Ccmudw== 2020-03-11T22:48:33.217 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDBamleYQ3FDBAAZ20T6KZqz6uf3QE6Ccmudw=="}' 2020-03-11T22:48:33.217 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3b65ff47-d96d-4084-819c-6d6d58dbdd82 -i td/osd-backfill-prio/4/new.json 2020-03-11T22:48:33.790 INFO:tasks.workunit.client.0.smithi016.stdout:4 2020-03-11T22:48:33.799 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-03-11T22:48:33.800 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDBamleYQ3FDBAAZ20T6KZqz6uf3QE6Ccmudw== --osd-uuid 3b65ff47-d96d-4084-819c-6d6d58dbdd82 2020-03-11T22:48:33.827 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:33.823+0000 7ff7385bbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:33.828 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:33.825+0000 7ff7385bbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:33.830 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:33.827+0000 7ff7385bbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:33.831 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:33.828+0000 7ff7385bbec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:33.831 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:33.828+0000 7ff7385bbec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:33.831 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:33.828+0000 7ff7385bbec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:33.885 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:33.882+0000 7ff7385bbec0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-03-11T22:48:37.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-03-11T22:48:37.067 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:37.068 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:48:37.068 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:37.068 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd4 key to auth repository 2020-03-11T22:48:37.559 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.4 2020-03-11T22:48:37.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:48:37.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:37.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:37.560 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:37.560 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=32b3f5bb-6a40-443f-838b-82b0ad2a0d82 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:37.584 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:37.580+0000 7f91e2e69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:37.612 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:37.609+0000 7f91e2e69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:37.614 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:37.611+0000 7f91e2e69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:37.928 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:48:37.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:37.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:48:37.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:37.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:37.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:37.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:37.930 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:37.930 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:48:37.930 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:48:38.134 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:38.130+0000 7f91e2e69ec0 -1 Falling back to public interface 2020-03-11T22:48:38.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:38.962 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:48:38.958+0000 7f91e2e69ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:48:39.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:39.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:39.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:39.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:39.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:48:39.299 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:48:39.706 INFO:tasks.workunit.client.0.smithi016.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64484,v1:127.0.0.1:6835/64484] [v2:127.0.0.1:6836/64484,v1:127.0.0.1:6837/64484] exists,up 3b65ff47-d96d-4084-819c-6d6d58dbdd82 2020-03-11T22:48:39.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:39.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:39.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:39.708 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-03-11T22:48:39.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:39.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-03-11T22:48:39.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:48:40.301 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test1' already exists 2020-03-11T22:48:40.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:41.310 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-03-11T22:48:42.143 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 1 size to 2 2020-03-11T22:48:42.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:42.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-03-11T22:48:42.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:48:42.828 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test2' already exists 2020-03-11T22:48:42.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:43.838 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-03-11T22:48:44.490 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 2 size to 2 2020-03-11T22:48:44.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:44.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-03-11T22:48:44.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:48:45.234 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test3' already exists 2020-03-11T22:48:45.242 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:46.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:48:47.156 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 3 size to 2 2020-03-11T22:48:47.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:47.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-03-11T22:48:47.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-03-11T22:48:47.737 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test4' already exists 2020-03-11T22:48:47.744 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:48.746 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-03-11T22:48:49.378 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 4 size to 2 2020-03-11T22:48:49.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:49.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-03-11T22:48:49.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-03-11T22:48:50.018 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test5' already exists 2020-03-11T22:48:50.026 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:51.028 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-03-11T22:48:51.786 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 5 size to 2 2020-03-11T22:48:51.794 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:51.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-03-11T22:48:51.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-03-11T22:48:52.316 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test6' already exists 2020-03-11T22:48:52.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:53.325 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-03-11T22:48:54.000 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 6 size to 2 2020-03-11T22:48:54.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:54.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-03-11T22:48:54.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-03-11T22:48:54.622 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test7' already exists 2020-03-11T22:48:54.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:55.633 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:48:56.404 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 7 size to 2 2020-03-11T22:48:56.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:56.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-03-11T22:48:56.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-03-11T22:48:57.145 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test8' already exists 2020-03-11T22:48:57.153 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:58.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-03-11T22:48:58.777 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 8 size to 2 2020-03-11T22:48:58.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:58.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-03-11T22:48:58.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-03-11T22:48:59.508 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test9' already exists 2020-03-11T22:48:59.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:00.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-03-11T22:49:01.126 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 9 size to 2 2020-03-11T22:49:01.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:01.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-03-11T22:49:01.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-03-11T22:49:01.640 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test10' already exists 2020-03-11T22:49:01.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:02.649 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-03-11T22:49:03.474 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 10 size to 2 2020-03-11T22:49:03.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-03-11T22:49:08.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-03-11T22:49:08.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:08.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:08.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:08.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:08.485 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:08.486 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:08.486 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:08.486 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:08.486 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:08.486 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:08.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:08.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:08.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:08.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:08.592 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:08.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:08.963 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:49:08.963 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:49:08.964 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:49:08.964 INFO:tasks.workunit.client.0.smithi016.stderr:4' 2020-03-11T22:49:08.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:08.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:08.964 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:09.100 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-11T22:49:09.100 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-11T22:49:09.100 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:09.100 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:09.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:49:09.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2020-03-11T22:49:09.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:09.233 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:09.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:49:09.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451' 2020-03-11T22:49:09.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:09.366 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:09.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345930 2020-03-11T22:49:09.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451 3-85899345930' 2020-03-11T22:49:09.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:09.500 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:09.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182408 2020-03-11T22:49:09.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451 3-85899345930 4-107374182408' 2020-03-11T22:49:09.632 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:09.633 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-11T22:49:09.633 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:09.634 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:09.634 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-11T22:49:09.634 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:09.635 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-11T22:49:09.635 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-11T22:49:09.635 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:09.636 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836494 2020-03-11T22:49:10.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836494 2020-03-11T22:49:10.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:11.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:11.006 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:11.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836494 2020-03-11T22:49:11.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:11.389 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:49:11.389 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:11.390 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:11.391 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:49:11.391 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:11.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:49:11.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:49:11.392 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:11.392 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:49:11.796 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672972 2020-03-11T22:49:11.797 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:11.797 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:49:11.797 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:11.802 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:11.802 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:49:11.802 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:11.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:49:11.803 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:49:11.803 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:11.803 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:49:12.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:49:12.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:12.167 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345930 2020-03-11T22:49:12.167 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:12.167 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:12.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345930 2020-03-11T22:49:12.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:12.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345930 2020-03-11T22:49:12.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2020-03-11T22:49:12.169 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:12.170 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345930 2020-03-11T22:49:12.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345930 -lt 85899345930 2020-03-11T22:49:12.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:12.535 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182408 2020-03-11T22:49:12.535 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:12.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:12.536 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182408 2020-03-11T22:49:12.536 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:12.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182408 2020-03-11T22:49:12.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2020-03-11T22:49:12.537 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182408 2020-03-11T22:49:12.537 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:12.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182409 -lt 107374182408 2020-03-11T22:49:12.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:49:12.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:12.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:13.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-03-11T22:49:13.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:13.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:13.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:13.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:13.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:13.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:13.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:13.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-03-11T22:49:13.763 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:13.764 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:13.764 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:14.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-03-11T22:49:14.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:14.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:14.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:49:14.588 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:14.589 INFO:tasks.workunit.client.0.smithi016.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:04.160489+0000 0'0 81:20 [2,1] 2 [2,1] 2 0'0 2020-03-11T22:49:01.459355+0000 0'0 2020-03-11T22:49:01.459355+0000 0 2020-03-11T22:49:14.589 INFO:tasks.workunit.client.0.smithi016.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:01.258692+0000 0'0 79:23 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:59.225646+0000 0'0 2020-03-11T22:48:59.225646+0000 0 2020-03-11T22:49:14.589 INFO:tasks.workunit.client.0.smithi016.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:59.600269+0000 0'0 79:29 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:48:56.850995+0000 0'0 2020-03-11T22:48:56.850995+0000 0 2020-03-11T22:49:14.589 INFO:tasks.workunit.client.0.smithi016.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:56.927064+0000 0'0 79:31 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:48:54.370142+0000 0'0 2020-03-11T22:48:54.370142+0000 0 2020-03-11T22:49:14.589 INFO:tasks.workunit.client.0.smithi016.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:45.540190+0000 0'0 79:52 [3,1] 3 [3,1] 3 0'0 2020-03-11T22:48:42.530584+0000 0'0 2020-03-11T22:48:42.530584+0000 0 2020-03-11T22:49:14.589 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:42.015654+0000 0'0 79:57 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:48:40.040132+0000 0'0 2020-03-11T22:48:40.040132+0000 0 2020-03-11T22:49:14.590 INFO:tasks.workunit.client.0.smithi016.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:47.648850+0000 0'0 79:48 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:44.827101+0000 0'0 2020-03-11T22:48:44.827101+0000 0 2020-03-11T22:49:14.590 INFO:tasks.workunit.client.0.smithi016.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:50.209601+0000 0'0 79:44 [4,0] 4 [4,0] 4 0'0 2020-03-11T22:48:47.501647+0000 0'0 2020-03-11T22:48:47.501647+0000 0 2020-03-11T22:49:14.590 INFO:tasks.workunit.client.0.smithi016.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:51.939079+0000 0'0 79:39 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:48:49.787430+0000 0'0 2020-03-11T22:48:49.787430+0000 0 2020-03-11T22:49:14.590 INFO:tasks.workunit.client.0.smithi016.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:54.262766+0000 0'0 79:35 [4,2] 4 [4,2] 4 0'0 2020-03-11T22:48:52.116624+0000 0'0 2020-03-11T22:48:52.116624+0000 0 2020-03-11T22:49:14.590 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:49:14.590 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:14.591 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:49:14.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-03-11T22:49:14.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-03-11T22:49:14.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-03-11T22:49:14.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-03-11T22:49:14.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-03-11T22:49:14.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-03-11T22:49:14.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-03-11T22:49:14.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-03-11T22:49:14.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-03-11T22:49:14.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-03-11T22:49:14.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-03-11T22:49:14.600 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-03-11T22:49:14.601 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:14.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:14.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-03-11T22:49:14.602 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:14.958 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:14.958 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:14.959 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:14.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:49:14.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-03-11T22:49:14.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-03-11T22:49:14.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:15.317 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:15.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:49:15.318 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:15.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-03-11T22:49:15.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:15.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-03-11T22:49:15.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-03-11T22:49:15.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:15.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-03-11T22:49:15.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:16.056 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:16.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:49:16.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:16.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:49:16.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:16.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-03-11T22:49:16.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-03-11T22:49:16.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:16.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-03-11T22:49:16.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:16.473 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:16.474 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:49:16.474 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:16.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:49:16.475 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:16.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-03-11T22:49:16.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:49:16.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:16.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-03-11T22:49:16.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:16.840 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:16.840 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:49:16.840 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:16.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:49:16.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:16.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-03-11T22:49:16.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-03-11T22:49:16.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:16.842 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-03-11T22:49:16.843 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:17.203 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:17.204 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:17.204 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:17.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:49:17.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:17.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-03-11T22:49:17.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:49:17.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-03-11T22:49:17.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-03-11T22:49:17.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-03-11T22:49:17.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-03-11T22:49:17.207 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:49:17.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-03-11T22:49:17.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:17.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:17.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-03-11T22:49:17.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:17.209 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-03-11T22:49:17.209 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-03-11T22:49:17.209 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:49:17.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:49:17.703 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test2' does not exist 2020-03-11T22:49:17.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:17.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-03-11T22:49:17.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:17.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-03-11T22:49:17.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-03-11T22:49:17.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-03-11T22:49:17.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-03-11T22:49:18.273 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test4' does not exist 2020-03-11T22:49:18.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:18.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-03-11T22:49:18.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-03-11T22:49:18.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-03-11T22:49:18.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-03-11T22:49:18.820 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test5' does not exist 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-03-11T22:49:19.358 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test6' does not exist 2020-03-11T22:49:19.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:19.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-03-11T22:49:19.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:19.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-03-11T22:49:19.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-03-11T22:49:19.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-03-11T22:49:19.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-03-11T22:49:19.902 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test8' does not exist 2020-03-11T22:49:19.910 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:19.910 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-03-11T22:49:19.911 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-03-11T22:49:19.911 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-03-11T22:49:19.911 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-03-11T22:49:20.483 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test9' does not exist 2020-03-11T22:49:20.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:20.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-03-11T22:49:20.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-03-11T22:49:20.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-03-11T22:49:20.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-03-11T22:49:21.038 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test10' does not exist 2020-03-11T22:49:21.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-03-11T22:49:21.804 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 3 size to 1 2020-03-11T22:49:21.812 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-03-11T22:49:22.726 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 7 size to 1 2020-03-11T22:49:22.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-03-11T22:49:22.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:22.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:22.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:22.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:22.736 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:22.737 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:22.737 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:22.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:22.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:22.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:22.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:22.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:22.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:22.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:22.834 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:23.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:23.205 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:49:23.205 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:49:23.206 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:49:23.206 INFO:tasks.workunit.client.0.smithi016.stderr:4' 2020-03-11T22:49:23.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:23.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:23.206 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:23.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836498 2020-03-11T22:49:23.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498' 2020-03-11T22:49:23.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:23.338 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:23.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672977 2020-03-11T22:49:23.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977' 2020-03-11T22:49:23.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:23.471 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:23.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509456 2020-03-11T22:49:23.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509456' 2020-03-11T22:49:23.603 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:23.604 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:23.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345935 2020-03-11T22:49:23.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509456 3-85899345935' 2020-03-11T22:49:23.830 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:23.830 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:23.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182413 2020-03-11T22:49:23.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509456 3-85899345935 4-107374182413' 2020-03-11T22:49:23.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:23.871 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836498 2020-03-11T22:49:23.871 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:23.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:23.871 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836498 2020-03-11T22:49:23.872 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:23.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836498 2020-03-11T22:49:23.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-03-11T22:49:23.873 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:23.873 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836498 2020-03-11T22:49:24.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836496 -lt 21474836498 2020-03-11T22:49:24.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:25.241 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:25.242 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:25.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836498 2020-03-11T22:49:25.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:25.621 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672977 2020-03-11T22:49:25.621 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:25.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:25.622 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672977 2020-03-11T22:49:25.622 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:25.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672977 2020-03-11T22:49:25.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2020-03-11T22:49:25.624 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:25.624 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672977 2020-03-11T22:49:26.017 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672977 -lt 42949672977 2020-03-11T22:49:26.017 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:26.017 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509456 2020-03-11T22:49:26.018 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:26.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:26.019 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509456 2020-03-11T22:49:26.019 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:26.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509456 2020-03-11T22:49:26.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2020-03-11T22:49:26.020 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:26.020 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509456 2020-03-11T22:49:26.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509456 -lt 64424509456 2020-03-11T22:49:26.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:26.389 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345935 2020-03-11T22:49:26.389 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:26.389 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:26.390 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345935 2020-03-11T22:49:26.390 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:26.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345935 2020-03-11T22:49:26.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345935' 2020-03-11T22:49:26.391 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:26.391 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345935 2020-03-11T22:49:26.782 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345935 -lt 85899345935 2020-03-11T22:49:26.782 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:26.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182413 2020-03-11T22:49:26.783 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:26.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:26.784 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182413 2020-03-11T22:49:26.784 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:26.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182413 2020-03-11T22:49:26.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2020-03-11T22:49:26.785 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:26.785 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182413 2020-03-11T22:49:27.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182414 -lt 107374182413 2020-03-11T22:49:27.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:49:27.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:27.188 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:27.663 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:49:27.663 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:27.663 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:27.663 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:27.663 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:27.664 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:27.664 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:27.664 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:28.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:49:28.025 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:28.026 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:28.026 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:28.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:49:28.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:28.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:28.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:49:28.552 INFO:tasks.workunit.client.0.smithi016.stderr:10+0 records in 2020-03-11T22:49:28.553 INFO:tasks.workunit.client.0.smithi016.stderr:10+0 records out 2020-03-11T22:49:28.553 INFO:tasks.workunit.client.0.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0462751 s, 227 MB/s 2020-03-11T22:49:28.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-03-11T22:49:28.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:49:28.554 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:49:28.554 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:28.554 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:49:28.988 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:28.988 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:49:29.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:29.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:49:29.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:29.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:49:30.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:30.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:49:30.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:30.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:49:31.504 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:31.504 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:49:31.928 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:31.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:49:32.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:32.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:49:32.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:32.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:49:33.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:33.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:49:33.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:33.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:49:34.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:34.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:49:34.743 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:34.744 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:49:35.222 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:35.222 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:49:35.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:35.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:49:36.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:36.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:49:36.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:36.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:49:37.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:37.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:49:37.663 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:37.663 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:49:38.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:38.119 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:49:38.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:38.569 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:49:39.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:39.062 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:49:39.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:39.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:49:40.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:40.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:49:40.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:40.621 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:49:41.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:41.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:49:41.545 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:41.545 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:49:42.083 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:42.083 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:49:42.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:42.554 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:49:43.116 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:43.117 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:49:43.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:43.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:49:44.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:44.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:49:44.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:44.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:49:44.957 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:44.957 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:49:45.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:45.438 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:49:45.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:45.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:49:46.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:46.372 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:49:46.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:46.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:49:47.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:49:47.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:49:48.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:48.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:49:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:49:49.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:49:49.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.707 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:49:50.197 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.197 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:49:50.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:49:51.261 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.261 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:49:51.680 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:49:52.231 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.231 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:49:52.681 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-03-11T22:49:52.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-03-11T22:49:52.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:49:52.682 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:49:52.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.683 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.032 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.033 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.592 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.841 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.841 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.509 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.509 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.084 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.335 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.934 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.934 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:49:58.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:49:58.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.427 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:49:58.731 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:49:59.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:49:59.306 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.309 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:49:59.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:49:59.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:50:00.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:50:00.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:50:00.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:50:01.078 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.078 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:50:01.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:50:01.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:50:01.866 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.866 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:50:02.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:50:02.458 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.458 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:50:02.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:50:03.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:50:03.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.367 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:50:03.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:50:03.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:50:04.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:50:04.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:50:04.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:50:05.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.009 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:50:05.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:50:05.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.560 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:50:05.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:50:06.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.118 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:50:06.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:50:06.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:50:06.977 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-03-11T22:50:06.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-03-11T22:50:06.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:50:06.978 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:50:06.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.399 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.400 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.419 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.203 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.203 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.570 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.570 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-03-11T22:50:11.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-03-11T22:50:11.479 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.479 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-03-11T22:50:11.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-03-11T22:50:12.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.046 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-03-11T22:50:12.362 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.363 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-03-11T22:50:12.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-03-11T22:50:12.957 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.957 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-03-11T22:50:13.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-03-11T22:50:13.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.500 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-03-11T22:50:13.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-03-11T22:50:14.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-03-11T22:50:14.313 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-03-11T22:50:14.572 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.572 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-03-11T22:50:14.913 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-03-11T22:50:15.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-03-11T22:50:15.481 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.481 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-03-11T22:50:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-03-11T22:50:16.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-03-11T22:50:16.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.281 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-03-11T22:50:16.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-03-11T22:50:16.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-03-11T22:50:17.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-03-11T22:50:17.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.477 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-03-11T22:50:17.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-03-11T22:50:18.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:18.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-03-11T22:50:18.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:18.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-03-11T22:50:18.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:18.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-03-11T22:50:18.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:18.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-03-11T22:50:19.232 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:19.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-03-11T22:50:19.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:19.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-03-11T22:50:19.799 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:19.799 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-03-11T22:50:20.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:20.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-03-11T22:50:20.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:20.291 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-03-11T22:50:20.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:20.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-03-11T22:50:20.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:20.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-03-11T22:50:21.091 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:21.091 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-03-11T22:50:21.370 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-03-11T22:50:21.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-03-11T22:50:21.371 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-03-11T22:50:21.372 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:50:21.372 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:50:21.373 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:50:21.373 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:50:21.373 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:50:21.373 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:50:21.373 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:50:21.754 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:50:21.755 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:50:21.755 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:50:22.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-03-11T22:50:22.150 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:22.517 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:22.517 INFO:tasks.workunit.client.0.smithi016.stdout:7.0 47 0 0 0 0 492830720 0 0 41 41 active+clean 2020-03-11T22:49:22.618732+0000 93'141 93:186 [1] 1 [1] 1 0'0 2020-03-11T22:48:54.370142+0000 0'0 2020-03-11T22:48:54.370142+0000 0 2020-03-11T22:50:22.517 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:48:42.015654+0000 93'150 93:214 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:48:40.040132+0000 0'0 2020-03-11T22:48:40.040132+0000 0 2020-03-11T22:50:22.517 INFO:tasks.workunit.client.0.smithi016.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:22.364467+0000 93'150 93:212 [1] 1 [1] 1 0'0 2020-03-11T22:48:44.827101+0000 0'0 2020-03-11T22:48:44.827101+0000 0 2020-03-11T22:50:22.517 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:50:22.518 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:22.518 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:50:22.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-03-11T22:50:22.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-03-11T22:50:23.141 INFO:tasks.workunit.client.0.smithi016.stderr:nobackfill is set 2020-03-11T22:50:23.149 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-03-11T22:50:23.933 INFO:tasks.workunit.client.0.smithi016.stderr:noout is set 2020-03-11T22:50:23.941 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:50:24.644 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 7 size to 2 2020-03-11T22:50:24.652 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-03-11T22:50:26.654 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:26.655 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:26.655 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:26.655 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:26.655 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:26.655 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:50:26.655 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:50:26.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:26.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:26.758 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:50:26.758 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:50:26.758 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:26.758 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:50:26.761 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:26.761 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:50:26.766 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:26.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:26.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:50:26.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:27.125 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2020-03-11T22:50:27.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:50:29.126 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:29.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:50:29.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:29.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 2 = 10 ']' 2020-03-11T22:50:29.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:50:31.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:31.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:50:31.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:31.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-03-11T22:50:31.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:31.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:31.872 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:32.241 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:32.242 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:50:32.242 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:50:32.242 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:50:32.242 INFO:tasks.workunit.client.0.smithi016.stderr:4' 2020-03-11T22:50:32.242 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:32.242 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:32.242 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:32.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836514 2020-03-11T22:50:32.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836514' 2020-03-11T22:50:32.407 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:32.408 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:32.540 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672993 2020-03-11T22:50:32.540 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836514 1-42949672993' 2020-03-11T22:50:32.540 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:32.540 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:32.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509472 2020-03-11T22:50:32.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472' 2020-03-11T22:50:32.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:32.673 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:32.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345950 2020-03-11T22:50:32.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-85899345950' 2020-03-11T22:50:32.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:32.816 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:32.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182429 2020-03-11T22:50:32.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509472 3-85899345950 4-107374182429' 2020-03-11T22:50:32.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:32.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836514 2020-03-11T22:50:32.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:32.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:32.950 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836514 2020-03-11T22:50:32.950 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836514 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836514 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:33.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836514 -lt 21474836514 2020-03-11T22:50:33.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:33.316 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672993 2020-03-11T22:50:33.317 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:33.317 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:33.318 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672993 2020-03-11T22:50:33.318 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:33.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672993 2020-03-11T22:50:33.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2020-03-11T22:50:33.319 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:33.319 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672993 2020-03-11T22:50:33.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672993 -lt 42949672993 2020-03-11T22:50:33.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:33.719 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509472 2020-03-11T22:50:33.719 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:33.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:33.720 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509472 2020-03-11T22:50:33.720 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:33.721 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509472 2020-03-11T22:50:33.721 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2020-03-11T22:50:33.721 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:33.722 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509472 2020-03-11T22:50:34.088 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509472 -lt 64424509472 2020-03-11T22:50:34.088 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:34.089 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345950 2020-03-11T22:50:34.089 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:34.090 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:34.090 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345950 2020-03-11T22:50:34.090 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:34.091 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345950 2020-03-11T22:50:34.091 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345950' 2020-03-11T22:50:34.092 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:34.092 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345950 2020-03-11T22:50:34.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345948 -lt 85899345950 2020-03-11T22:50:34.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:35.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:35.495 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:35.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345951 -lt 85899345950 2020-03-11T22:50:35.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:35.878 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182429 2020-03-11T22:50:35.878 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:35.878 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:35.879 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182429 2020-03-11T22:50:35.879 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:35.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182429 2020-03-11T22:50:35.880 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182429' 2020-03-11T22:50:35.880 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:35.880 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182429 2020-03-11T22:50:36.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182429 -lt 107374182429 2020-03-11T22:50:36.269 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:36.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:36.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:36.271 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:36.271 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:36.271 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:50:36.271 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:50:36.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:36.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:36.362 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:50:36.362 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:50:36.363 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:36.363 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:36.363 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:50:36.363 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:50:36.363 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:36.363 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:50:36.363 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:50:36.364 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:36.364 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:36.364 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:36.364 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:50:36.364 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:50:36.364 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:36.364 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:36.365 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:50:36.365 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:50:36.366 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:36.366 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:50:36.369 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-03-11T22:50:36.864 INFO:tasks.workunit.client.0.smithi016.stderr:osd.0 is already out. 2020-03-11T22:50:36.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-03-11T22:50:38.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:38.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:38.873 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:39.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:39.254 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:50:39.255 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:50:39.255 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:50:39.255 INFO:tasks.workunit.client.0.smithi016.stderr:4' 2020-03-11T22:50:39.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:39.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:39.255 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:39.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836518 2020-03-11T22:50:39.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518' 2020-03-11T22:50:39.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:39.387 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:39.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672996 2020-03-11T22:50:39.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518 1-42949672996' 2020-03-11T22:50:39.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:39.520 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:39.652 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509475 2020-03-11T22:50:39.652 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518 1-42949672996 2-64424509475' 2020-03-11T22:50:39.652 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:39.652 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:39.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345954 2020-03-11T22:50:39.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518 1-42949672996 2-64424509475 3-85899345954' 2020-03-11T22:50:39.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:39.784 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:39.916 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182432 2020-03-11T22:50:39.916 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836518 1-42949672996 2-64424509475 3-85899345954 4-107374182432' 2020-03-11T22:50:39.917 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:39.917 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836518 2020-03-11T22:50:39.917 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:39.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:39.918 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836518 2020-03-11T22:50:39.918 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:39.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836518 2020-03-11T22:50:39.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2020-03-11T22:50:39.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:39.920 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836518 2020-03-11T22:50:40.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836518 2020-03-11T22:50:40.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:41.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:41.286 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:41.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836518 -lt 21474836518 2020-03-11T22:50:41.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:41.670 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672996 2020-03-11T22:50:41.670 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:41.671 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:41.671 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672996 2020-03-11T22:50:41.671 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:41.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672996 2020-03-11T22:50:41.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2020-03-11T22:50:41.672 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:41.673 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672996 2020-03-11T22:50:42.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672997 -lt 42949672996 2020-03-11T22:50:42.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:42.143 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509475 2020-03-11T22:50:42.143 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:42.144 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:42.144 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509475 2020-03-11T22:50:42.144 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:42.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509475 2020-03-11T22:50:42.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2020-03-11T22:50:42.146 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509475 2020-03-11T22:50:42.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:42.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509475 -lt 64424509475 2020-03-11T22:50:42.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:42.520 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345954 2020-03-11T22:50:42.520 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:42.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:42.521 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345954 2020-03-11T22:50:42.521 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:42.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345954 2020-03-11T22:50:42.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345954' 2020-03-11T22:50:42.522 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:42.522 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345954 2020-03-11T22:50:42.902 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345954 -lt 85899345954 2020-03-11T22:50:42.902 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:42.902 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182432 2020-03-11T22:50:42.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:42.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:42.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182432 2020-03-11T22:50:42.904 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:42.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182432 2020-03-11T22:50:42.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182432' 2020-03-11T22:50:42.905 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:42.905 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182432 2020-03-11T22:50:43.270 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182433 -lt 107374182432 2020-03-11T22:50:43.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:43.271 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:43.271 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:50:43.272 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:50:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:43.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:43.363 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:50:43.364 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:50:43.364 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:43.364 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:43.364 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:50:43.364 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:43.364 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:50:43.365 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:50:43.365 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:43.365 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:50:43.365 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:50:43.365 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:43.365 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:43.365 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:43.366 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:43.366 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:50:43.366 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:50:43.366 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:43.366 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:50:43.366 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:50:43.366 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:43.366 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:43.367 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:43.367 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:50:43.367 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:50:43.367 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:43.367 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:43.367 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:50:43.367 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:50:43.368 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:43.368 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:50:43.370 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:43.712 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:43.713 INFO:tasks.workunit.client.0.smithi016.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:50:31.759226+0000 93'150 104:213 [1,4] 1 [1] 1 0'0 2020-03-11T22:48:54.370142+0000 0'0 2020-03-11T22:48:54.370142+0000 0 2020-03-11T22:50:43.713 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:50:37.962066+0000 93'150 105:230 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:48:40.040132+0000 0'0 2020-03-11T22:48:40.040132+0000 0 2020-03-11T22:50:43.713 INFO:tasks.workunit.client.0.smithi016.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:22.364467+0000 93'150 104:220 [1] 1 [1] 1 0'0 2020-03-11T22:48:44.827101+0000 0'0 2020-03-11T22:48:44.827101+0000 0 2020-03-11T22:50:43.713 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:50:43.713 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:43.714 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:50:43.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:50:44.388 INFO:tasks.workunit.client.0.smithi016.stderr:set pool 3 size to 2 2020-03-11T22:50:44.396 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-03-11T22:50:46.397 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:46.398 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:46.398 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:46.782 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:46.783 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:50:46.783 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:50:46.783 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:50:46.783 INFO:tasks.workunit.client.0.smithi016.stderr:4' 2020-03-11T22:50:46.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:46.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:46.784 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:46.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836521 2020-03-11T22:50:46.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521' 2020-03-11T22:50:46.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:46.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:47.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673000 2020-03-11T22:50:47.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521 1-42949673000' 2020-03-11T22:50:47.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:47.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:47.190 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509478 2020-03-11T22:50:47.190 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521 1-42949673000 2-64424509478' 2020-03-11T22:50:47.190 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:47.191 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:47.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345957 2020-03-11T22:50:47.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521 1-42949673000 2-64424509478 3-85899345957' 2020-03-11T22:50:47.326 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:47.327 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:47.458 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182436 2020-03-11T22:50:47.459 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521 1-42949673000 2-64424509478 3-85899345957 4-107374182436' 2020-03-11T22:50:47.459 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:47.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836521 2020-03-11T22:50:47.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:47.460 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:47.460 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836521 2020-03-11T22:50:47.460 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:47.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836521 2020-03-11T22:50:47.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2020-03-11T22:50:47.461 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836521 2020-03-11T22:50:47.461 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:47.827 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836521 2020-03-11T22:50:47.827 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:48.829 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:49.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836521 -lt 21474836521 2020-03-11T22:50:49.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:49.211 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673000 2020-03-11T22:50:49.211 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:49.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:49.212 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673000 2020-03-11T22:50:49.212 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:49.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673000 2020-03-11T22:50:49.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2020-03-11T22:50:49.213 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:49.213 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949673000 2020-03-11T22:50:49.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673000 -lt 42949673000 2020-03-11T22:50:49.732 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:49.733 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509478 2020-03-11T22:50:49.733 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:49.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:49.734 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509478 2020-03-11T22:50:49.734 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:49.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509478 2020-03-11T22:50:49.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2020-03-11T22:50:49.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:49.735 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509478 2020-03-11T22:50:50.103 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509479 -lt 64424509478 2020-03-11T22:50:50.103 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:50.104 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345957 2020-03-11T22:50:50.104 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:50.104 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:50.105 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345957 2020-03-11T22:50:50.105 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:50.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345957 2020-03-11T22:50:50.106 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345957' 2020-03-11T22:50:50.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:50.106 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345957 2020-03-11T22:50:50.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345957 -lt 85899345957 2020-03-11T22:50:50.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:50.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182436 2020-03-11T22:50:50.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:50.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:50.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182436 2020-03-11T22:50:50.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:50.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182436 2020-03-11T22:50:50.499 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182436' 2020-03-11T22:50:50.500 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:50.500 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182436 2020-03-11T22:50:50.940 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182437 -lt 107374182436 2020-03-11T22:50:50.940 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:50.940 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:50.941 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:50.941 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:50.942 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:50.942 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:50:50.942 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:50:50.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:50.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:51.041 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:51.042 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:50:51.042 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:50:51.042 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:51.042 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:51.042 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:50:51.042 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:51.042 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:50:51.043 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:50:51.043 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:51.043 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:50:51.043 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:50:51.043 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:51.043 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:51.043 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:50:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:50:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:50:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:50:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:50:51.044 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:51.045 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:51.045 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:51.045 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:51.045 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:50:51.045 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:50:51.045 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:51.046 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:50:51.046 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:50:51.046 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:51.046 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:51.046 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:51.046 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:50:51.046 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:50:51.047 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:51.047 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:51.047 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:50:51.047 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:50:51.047 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:51.047 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:50:51.048 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:51.396 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:51.396 INFO:tasks.workunit.client.0.smithi016.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:50:31.759226+0000 93'150 108:216 [1,4] 1 [1] 1 0'0 2020-03-11T22:48:54.370142+0000 0'0 2020-03-11T22:48:54.370142+0000 0 2020-03-11T22:50:51.396 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:50:37.962066+0000 93'150 108:233 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:48:40.040132+0000 0'0 2020-03-11T22:48:40.040132+0000 0 2020-03-11T22:50:51.396 INFO:tasks.workunit.client.0.smithi016.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2020-03-11T22:50:45.419857+0000 93'150 109:230 [1,2] 1 [1] 1 0'0 2020-03-11T22:48:44.827101+0000 0'0 2020-03-11T22:48:44.827101+0000 0 2020-03-11T22:50:51.396 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:50:51.397 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:51.397 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:50:51.406 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:51.406 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-03-11T22:50:51.408 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-03-11T22:50:51.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-03-11T22:50:51.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:51.409 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:50:51.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:50:51.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:50:51.412 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:50:51.412 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:51.412 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:50:51.414 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:51.414 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:51.415 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:51.415 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:51.415 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:50:51.416 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:51.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-03-11T22:50:51.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-03-11T22:50:53.774 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:53.775 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:53.775 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:53.775 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:53.775 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:53.776 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:50:53.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:50:53.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:53.776 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:53.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:53.887 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:50:53.887 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:50:53.887 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:53.887 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:53.887 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:50:53.888 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:53.888 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:50:53.888 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:50:53.888 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:53.889 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:50:53.889 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:50:53.889 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:53.889 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:53.890 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:53.890 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:50:53.890 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:53.890 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 254, 2020-03-11T22:50:53.891 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:50:53.891 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:53.891 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:50:53.891 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:50:53.892 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:53.892 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:53.892 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:53.892 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:53.892 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:50:53.893 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:50:53.893 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:50:53.893 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:50:53.893 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:50:53.893 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:50:53.894 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:53.894 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:50:53.894 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:50:53.894 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:50:53.894 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:50:53.895 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:50:53.895 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:50:53.895 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:50:53.895 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:50:53.895 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:50:53.896 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:53.897 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:50:53.897 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:53.897 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:53.897 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:53.897 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:53.897 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:54.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:54.260 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:50:54.260 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:50:54.260 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:50:54.260 INFO:tasks.workunit.client.0.smithi016.stderr:4' 2020-03-11T22:50:54.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:54.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:54.260 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:54.391 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836525 2020-03-11T22:50:54.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836525' 2020-03-11T22:50:54.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:54.392 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:54.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673003 2020-03-11T22:50:54.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836525 1-42949673003' 2020-03-11T22:50:54.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:54.524 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:54.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509482 2020-03-11T22:50:54.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509482' 2020-03-11T22:50:54.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:54.657 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:54.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345961 2020-03-11T22:50:54.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509482 3-85899345961' 2020-03-11T22:50:54.789 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:54.789 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:54.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182439 2020-03-11T22:50:54.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509482 3-85899345961 4-107374182439' 2020-03-11T22:50:54.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:54.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836525 2020-03-11T22:50:54.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:54.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:54.922 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836525 2020-03-11T22:50:54.922 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:54.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836525 2020-03-11T22:50:54.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2020-03-11T22:50:54.924 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:54.924 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836525 2020-03-11T22:50:55.293 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836525 -lt 21474836525 2020-03-11T22:50:55.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:55.294 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673003 2020-03-11T22:50:55.294 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:55.295 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:55.295 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673003 2020-03-11T22:50:55.295 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:55.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673003 2020-03-11T22:50:55.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2020-03-11T22:50:55.297 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949673003 2020-03-11T22:50:55.297 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:55.760 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673003 -lt 42949673003 2020-03-11T22:50:55.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:55.762 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509482 2020-03-11T22:50:55.762 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:55.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:55.763 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509482 2020-03-11T22:50:55.763 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:55.764 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509482 2020-03-11T22:50:55.764 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2020-03-11T22:50:55.764 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:55.764 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509482 2020-03-11T22:50:56.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509482 -lt 64424509482 2020-03-11T22:50:56.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:56.145 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345961 2020-03-11T22:50:56.145 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:56.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:56.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345961 2020-03-11T22:50:56.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:56.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345961 2020-03-11T22:50:56.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345961' 2020-03-11T22:50:56.147 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:56.147 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345961 2020-03-11T22:50:56.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345959 -lt 85899345961 2020-03-11T22:50:56.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:57.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:57.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:57.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345961 -lt 85899345961 2020-03-11T22:50:57.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:57.893 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182439 2020-03-11T22:50:57.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:57.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:57.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182439 2020-03-11T22:50:57.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:57.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182439 2020-03-11T22:50:57.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2020-03-11T22:50:57.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:57.895 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182439 2020-03-11T22:50:58.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182440 -lt 107374182439 2020-03-11T22:50:58.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-03-11T22:50:58.638 INFO:tasks.workunit.client.0.smithi016.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:50:58.638 INFO:tasks.workunit.client.0.smithi016.stderr: 2020-03-11T22:50:58.645 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-03-11T22:51:00.648 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:00.648 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:00.648 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:00.648 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:00.649 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:00.649 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:51:00.649 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:51:00.649 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:00.650 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:00.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:00.760 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:00.760 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:00.760 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:00.760 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:00.760 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:00.761 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:00.761 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:00.761 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:00.761 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:00.761 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:00.761 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:00.761 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:00.762 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:00.762 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:00.762 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:00.762 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:00.762 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:00.762 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:00.762 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:00.763 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:00.763 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:00.763 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:00.763 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:00.763 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:00.763 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:00.763 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:00.764 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:00.764 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:00.764 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:00.764 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:00.764 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:00.764 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:00.764 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:00.765 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:00.765 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:00.765 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:00.765 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:00.765 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:00.765 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:00.765 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:00.765 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:00.767 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:00.767 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-03-11T22:51:00.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-03-11T22:51:00.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:51:00.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:00.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:51:00.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-03-11T22:51:00.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-03-11T22:51:00.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-03-11T22:51:00.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:00.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:51:00.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-03-11T22:51:00.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:51:00.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-03-11T22:51:01.122 INFO:tasks.workunit.client.0.smithi016.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:51:01.122 INFO:tasks.workunit.client.0.smithi016.stderr: 2020-03-11T22:51:01.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-03-11T22:51:06.132 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:06.133 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:06.133 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:06.133 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:06.133 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:06.133 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:51:06.134 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:51:06.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:06.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:06.235 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:06.235 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:06.235 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:06.236 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:06.236 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:06.236 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:06.236 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:06.236 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:06.236 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:06.236 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:06.237 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:06.237 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:06.237 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:06.237 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:06.237 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:06.237 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:06.237 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:06.238 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:06.238 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:06.238 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:06.238 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:06.238 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:06.238 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:06.238 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:06.239 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:06.239 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:06.239 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:06.239 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:06.239 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:06.239 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:06.239 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:06.240 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:06.240 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:06.240 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:06.240 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:06.240 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:06.240 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:06.240 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:06.241 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:06.241 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:06.241 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:06.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:06.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:06.243 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:06.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:06.611 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:51:06.611 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:51:06.612 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:51:06.612 INFO:tasks.workunit.client.0.smithi016.stderr:4' 2020-03-11T22:51:06.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:06.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:06.612 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:06.743 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836529 2020-03-11T22:51:06.743 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529' 2020-03-11T22:51:06.743 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:06.744 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:06.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673008 2020-03-11T22:51:06.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529 1-42949673008' 2020-03-11T22:51:06.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:06.877 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:07.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509486 2020-03-11T22:51:07.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509486' 2020-03-11T22:51:07.019 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:07.019 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:07.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345965 2020-03-11T22:51:07.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509486 3-85899345965' 2020-03-11T22:51:07.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:07.141 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:07.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182444 2020-03-11T22:51:07.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509486 3-85899345965 4-107374182444' 2020-03-11T22:51:07.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:07.274 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836529 2020-03-11T22:51:07.274 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:07.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:07.275 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836529 2020-03-11T22:51:07.275 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:07.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836529 2020-03-11T22:51:07.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2020-03-11T22:51:07.276 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:07.276 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836529 2020-03-11T22:51:07.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836527 -lt 21474836529 2020-03-11T22:51:07.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:08.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:08.645 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:09.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836529 -lt 21474836529 2020-03-11T22:51:09.024 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:09.025 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673008 2020-03-11T22:51:09.025 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:09.025 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:09.027 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673008 2020-03-11T22:51:09.027 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:09.027 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673008 2020-03-11T22:51:09.027 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2020-03-11T22:51:09.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:09.028 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949673008 2020-03-11T22:51:09.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673008 -lt 42949673008 2020-03-11T22:51:09.392 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:09.392 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509486 2020-03-11T22:51:09.392 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:09.393 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:09.393 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509486 2020-03-11T22:51:09.394 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:09.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509486 2020-03-11T22:51:09.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509486' 2020-03-11T22:51:09.395 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:09.395 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509486 2020-03-11T22:51:09.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509487 -lt 64424509486 2020-03-11T22:51:09.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:09.760 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345965 2020-03-11T22:51:09.760 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:09.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:09.761 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345965 2020-03-11T22:51:09.761 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:09.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345965 2020-03-11T22:51:09.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345965' 2020-03-11T22:51:09.762 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:09.763 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345965 2020-03-11T22:51:10.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345965 -lt 85899345965 2020-03-11T22:51:10.128 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:10.129 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182444 2020-03-11T22:51:10.129 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:10.129 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:10.130 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182444 2020-03-11T22:51:10.130 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:10.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182444 2020-03-11T22:51:10.131 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2020-03-11T22:51:10.131 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:10.131 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182444 2020-03-11T22:51:10.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182444 -lt 107374182444 2020-03-11T22:51:10.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:51:10.851 INFO:tasks.workunit.client.0.smithi016.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-03-11T22:51:10.851 INFO:tasks.workunit.client.0.smithi016.stderr: 2020-03-11T22:51:10.859 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-03-11T22:51:12.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:12.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:12.862 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:12.862 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:12.862 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:12.862 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:51:12.862 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:51:12.863 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:12.863 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:12.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:12.975 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:12.975 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:12.975 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:12.975 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:12.975 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:12.975 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:12.976 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:12.976 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:12.976 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:12.976 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:12.976 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:12.976 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:12.976 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:12.977 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:12.977 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:12.977 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:12.977 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:12.977 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:12.977 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:12.977 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:12.978 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:12.978 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:12.978 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:12.978 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:12.978 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:12.979 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:12.979 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:12.979 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:12.979 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:12.979 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:12.979 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:12.979 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:12.980 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:12.980 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:12.980 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:12.980 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:12.980 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:12.980 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:12.980 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:12.981 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:12.981 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:12.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:12.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:51:12.982 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-03-11T22:51:12.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:51:12.983 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:12.983 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:51:12.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:51:12.983 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:51:12.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:51:12.983 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:12.984 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:51:12.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-03-11T22:51:12.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:51:12.984 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-03-11T22:51:13.632 INFO:tasks.workunit.client.0.smithi016.stderr:noout is unset 2020-03-11T22:51:13.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-03-11T22:51:14.426 INFO:tasks.workunit.client.0.smithi016.stderr:nobackfill is unset 2020-03-11T22:51:14.437 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:14.437 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:14.437 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:14.438 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:14.438 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:14.438 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:51:14.438 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:51:14.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:51:14.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:51:14.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:51:14.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:51:14.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:51:14.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:51:14.440 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:51:14.440 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:51:14.440 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:51:14.441 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:51:14.443 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:51:14.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:51:14.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:51:14.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:51:14.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:14.532 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:14.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:14.901 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:51:14.901 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:51:14.901 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:51:14.901 INFO:tasks.workunit.client.0.smithi016.stderr:4' 2020-03-11T22:51:14.902 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:14.902 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:14.902 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:15.094 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836533 2020-03-11T22:51:15.094 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836533' 2020-03-11T22:51:15.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:15.095 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:15.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673011 2020-03-11T22:51:15.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836533 1-42949673011' 2020-03-11T22:51:15.227 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:15.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:15.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509490 2020-03-11T22:51:15.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836533 1-42949673011 2-64424509490' 2020-03-11T22:51:15.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:15.359 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:15.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345969 2020-03-11T22:51:15.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836533 1-42949673011 2-64424509490 3-85899345969' 2020-03-11T22:51:15.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:15.493 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:15.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182448 2020-03-11T22:51:15.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836533 1-42949673011 2-64424509490 3-85899345969 4-107374182448' 2020-03-11T22:51:15.626 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:15.626 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836533 2020-03-11T22:51:15.627 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:15.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:15.627 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836533 2020-03-11T22:51:15.627 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:15.628 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836533 2020-03-11T22:51:15.628 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2020-03-11T22:51:15.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:15.629 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836533 2020-03-11T22:51:16.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836533 2020-03-11T22:51:16.020 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:17.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:17.021 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:17.444 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836533 -lt 21474836533 2020-03-11T22:51:17.444 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:17.445 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673011 2020-03-11T22:51:17.445 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:17.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:17.446 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673011 2020-03-11T22:51:17.447 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:17.447 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673011 2020-03-11T22:51:17.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2020-03-11T22:51:17.448 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:17.448 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949673011 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673012 -lt 42949673011 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509490 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:17.909 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:17.909 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509490 2020-03-11T22:51:17.909 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:17.910 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509490 2020-03-11T22:51:17.910 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2020-03-11T22:51:17.910 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:17.911 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509490 2020-03-11T22:51:18.356 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509490 -lt 64424509490 2020-03-11T22:51:18.356 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:18.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345969 2020-03-11T22:51:18.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:18.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:18.358 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345969 2020-03-11T22:51:18.358 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:18.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345969 2020-03-11T22:51:18.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345969' 2020-03-11T22:51:18.360 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:18.360 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345969 2020-03-11T22:51:18.807 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345969 -lt 85899345969 2020-03-11T22:51:18.807 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:18.807 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182448 2020-03-11T22:51:18.808 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:18.808 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:18.809 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182448 2020-03-11T22:51:18.809 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:18.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182448 2020-03-11T22:51:18.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2020-03-11T22:51:18.810 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:18.810 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182448 2020-03-11T22:51:19.207 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182448 -lt 107374182448 2020-03-11T22:51:19.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:51:19.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:19.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:19.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:51:19.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:19.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:19.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:19.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:19.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:19.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:19.771 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:20.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:20.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:20.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:20.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:20.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:20.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:51:20.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:20.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:51:20.654 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:20.655 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:20.655 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:20.746 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:20.746 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:20.746 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:20.746 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:20.747 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:20.747 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:20.747 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:20.747 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:20.747 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:20.747 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:20.747 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:20.747 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:20.748 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:20.748 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:20.748 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:20.748 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:20.748 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:20.748 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:20.748 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:20.749 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:20.749 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:20.749 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:20.749 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:20.749 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:20.749 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:20.749 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:20.749 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:20.750 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:20.750 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:20.750 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:20.750 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:20.750 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:20.751 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:20.751 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:20.751 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:20.751 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:20.751 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:20.751 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:20.751 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:20.751 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:20.752 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:20.753 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:20.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:20.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:20.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:20.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:20.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:20.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:20.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:20.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:21.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:21.274 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:21.275 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:21.275 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:21.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:21.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:21.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:21.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:21.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:21.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:21.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:21.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:21.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:22.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17771764 2020-03-11T22:51:22.233 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17771764 '!=' null 2020-03-11T22:51:22.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:22.234 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:22.234 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:22.234 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:22.326 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:22.326 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:22.326 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:22.326 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:22.327 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:22.327 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:22.327 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:22.327 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:22.327 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:22.327 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:22.327 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:22.328 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:22.328 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:22.328 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:22.328 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:22.328 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:22.328 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:22.328 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:22.329 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:22.329 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:22.329 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:22.329 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:22.329 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:22.329 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:22.329 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:22.330 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:22.330 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:22.330 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:22.330 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:22.330 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:22.330 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:22.330 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:22.331 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:22.331 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:22.331 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:22.332 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:22.332 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:22.332 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:22.332 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:22.332 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:22.332 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:22.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:22.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:22.435 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:22.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:22.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:22.435 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:22.436 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:22.436 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:22.436 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:22.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:22.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:22.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:22.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:23.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:23.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:23.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:23.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:23.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:23.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:23.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:23.298 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:23.298 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:23.811 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14609352 2020-03-11T22:51:23.812 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14609352 '!=' null 2020-03-11T22:51:23.812 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:23.812 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:23.812 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:23.812 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:23.903 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:23.903 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:23.904 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:23.904 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:23.904 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:23.904 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:23.904 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:23.904 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:23.904 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:23.905 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:23.905 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:23.905 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:23.905 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:23.905 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:23.905 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:23.905 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:23.906 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:23.906 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:23.906 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:23.906 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:23.906 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:23.906 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:23.906 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:23.908 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:23.908 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:23.908 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:23.908 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:23.908 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:23.908 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:23.908 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:23.909 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:23.909 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:23.909 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:23.909 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:23.910 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:24.011 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:24.012 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:24.012 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:24.013 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:24.013 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:24.013 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:24.013 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:24.013 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:24.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:24.372 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:24.372 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:24.372 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:24.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:24.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:24.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:24.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:24.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:24.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:24.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:24.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:24.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:25.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=12232133 2020-03-11T22:51:25.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 12232133 '!=' null 2020-03-11T22:51:25.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:25.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:25.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:25.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:25.548 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:25.549 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:25.549 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:25.549 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:25.549 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:25.549 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:25.549 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:25.550 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:25.550 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:25.550 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:25.550 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:25.550 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:25.550 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:25.550 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:25.550 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:25.551 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:25.551 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:25.551 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:25.551 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:25.551 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:25.551 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:25.551 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:25.552 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:25.552 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:25.552 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:25.552 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:25.552 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:25.552 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:25.552 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:25.552 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:25.553 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:25.553 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:25.553 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:25.553 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:25.553 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:25.553 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:25.554 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:25.554 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:25.554 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:25.554 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:25.554 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:25.556 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:25.658 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:25.658 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:25.658 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:25.658 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:25.658 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:25.659 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:25.659 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:25.659 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:26.015 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:26.015 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:26.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:26.016 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:26.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:26.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:26.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:26.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:26.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:26.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:26.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:26.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:26.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:27.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967570 2020-03-11T22:51:27.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967570 '!=' null 2020-03-11T22:51:27.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:27.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:27.037 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:27.038 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:27.129 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:27.129 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:27.130 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:27.130 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:27.130 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:27.130 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:27.130 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:27.130 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:27.130 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:27.131 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:27.132 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:27.133 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:27.134 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:27.135 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:27.136 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:27.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:27.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:27.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:27.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:27.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:27.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:27.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:27.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:27.684 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:27.685 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:27.685 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:27.685 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:28.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:28.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:28.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:28.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:28.178 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:28.178 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:28.178 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:28.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:28.178 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:28.723 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967570 2020-03-11T22:51:28.723 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967570 '!=' null 2020-03-11T22:51:28.723 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:28.723 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:28.723 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:28.723 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:28.816 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:28.816 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:28.816 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:28.816 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:28.816 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:28.817 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:28.817 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:28.817 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:28.817 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:28.817 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:28.818 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:28.818 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:28.818 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:28.818 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:28.818 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:28.818 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:28.819 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:28.819 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:28.819 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:28.819 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:28.819 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:28.819 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:28.819 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:28.820 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:28.820 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:28.820 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:28.820 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:28.820 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:28.820 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:28.820 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:28.821 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:28.821 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:28.821 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:28.821 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:28.821 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:28.821 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:28.821 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:28.822 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:28.822 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:28.822 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:28.822 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:28.823 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:28.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:28.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:28.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:28.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:28.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:28.926 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:28.926 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:28.926 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:29.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:29.285 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:29.286 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:29.286 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:29.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:29.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:29.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:29.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:29.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:29.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:29.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:29.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:29.768 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:30.398 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967548 2020-03-11T22:51:30.398 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967548 '!=' null 2020-03-11T22:51:30.399 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:30.399 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:30.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:30.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:30.491 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:30.491 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:30.491 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:30.491 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:30.492 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:30.492 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:30.492 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:30.492 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:30.492 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:30.492 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:30.492 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:30.492 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:30.493 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:30.493 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:30.493 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:30.493 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:30.493 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:30.493 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:30.494 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:30.494 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:30.494 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:30.494 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:30.494 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:30.494 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:30.495 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:30.495 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:30.495 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:30.495 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:30.495 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:30.495 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:30.495 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:30.495 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:30.496 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:30.496 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:30.496 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:30.496 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:30.496 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:30.496 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:30.496 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:30.497 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:30.497 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:30.498 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:30.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:30.599 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:30.600 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:30.600 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:30.600 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:30.600 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:30.601 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:30.601 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:30.958 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:30.959 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:30.959 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:30.959 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:31.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:31.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:31.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:31.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:31.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:31.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:31.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:31.434 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:31.434 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:32.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32325472 2020-03-11T22:51:32.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32325472 '!=' null 2020-03-11T22:51:32.036 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:32.036 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:32.036 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:32.036 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:32.129 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:32.129 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:32.129 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:32.129 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:32.129 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:32.129 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:32.130 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:32.130 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:32.130 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:32.130 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:32.130 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:32.130 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:32.130 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:32.131 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:32.131 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:32.131 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:32.131 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:32.131 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:32.131 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:32.131 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:32.131 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:32.132 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:32.132 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:32.132 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:32.132 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:32.132 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:32.132 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:32.136 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:32.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:32.238 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:32.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:32.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:32.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:32.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:32.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:32.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:32.597 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:32.597 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:32.598 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:32.598 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:33.081 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:33.081 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:33.081 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:33.649 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841551 2020-03-11T22:51:33.650 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841551 '!=' null 2020-03-11T22:51:33.650 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:33.650 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:33.650 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:33.650 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:33.741 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:33.741 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:33.741 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:33.742 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:33.742 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:33.742 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:33.742 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:33.742 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:33.742 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:33.742 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:33.743 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:33.743 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:33.743 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:33.743 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:33.743 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:33.743 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:33.743 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:33.743 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:33.744 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:33.744 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:33.744 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:33.744 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:33.744 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:33.744 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:33.744 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:33.745 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:33.745 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:33.745 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:33.745 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:33.745 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:33.745 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:33.745 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:33.746 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:33.746 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:33.746 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:33.746 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:33.746 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:33.746 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:33.746 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:33.746 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:33.747 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:33.748 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:33.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:33.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:33.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:33.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:33.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:33.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:33.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:33.851 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:34.208 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:34.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:34.208 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:34.209 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:34.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:34.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:34.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:34.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:34.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:34.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:34.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:34.689 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:34.689 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:35.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841510 2020-03-11T22:51:35.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841510 '!=' null 2020-03-11T22:51:35.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:35.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:35.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:35.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:35.349 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:35.349 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:35.349 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:35.349 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:35.350 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:35.350 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:35.350 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:35.350 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:35.350 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:35.350 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:35.351 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:35.351 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:35.351 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:35.351 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:35.351 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:35.351 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:35.351 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 151, 2020-03-11T22:51:35.351 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:35.352 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:35.352 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:35.352 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:35.352 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:35.352 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:35.352 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:35.352 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:35.353 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:35.353 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:35.353 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:35.353 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "7.0", 2020-03-11T22:51:35.353 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 254, 2020-03-11T22:51:35.353 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:35.353 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:35.354 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:35.354 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:35.354 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:35.354 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:35.354 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:35.354 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:35.354 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:35.355 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:35.355 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:35.356 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:35.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:35.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:35.458 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:35.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:35.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:35.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:35.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:35.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:35.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:35.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:35.818 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:35.818 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:36.469 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:36.469 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:36.469 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:36.469 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:36.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:36.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:36.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:36.470 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:36.470 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:36.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841510 2020-03-11T22:51:36.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841510 '!=' null 2020-03-11T22:51:36.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:36.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:36.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:36.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:37.047 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:37.048 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:37.048 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:37.048 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:37.048 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:37.048 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:37.048 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:37.049 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:37.049 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:37.049 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:37.049 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:37.049 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:37.049 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:37.049 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:37.050 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:37.050 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:37.050 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:37.050 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:37.050 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:37.050 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:37.050 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:37.050 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:37.052 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:37.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:37.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:37.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:37.157 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:37.157 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:37.158 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:37.158 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:37.158 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:37.158 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:37.517 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:37.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:37.518 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:37.518 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:38.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32762232 2020-03-11T22:51:38.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32762232 '!=' null 2020-03-11T22:51:38.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:38.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:38.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:38.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:38.607 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:38.607 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:38.607 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:38.608 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:38.608 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:38.608 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:38.608 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:38.608 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:38.608 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:38.609 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:38.609 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:38.609 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:38.609 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:38.609 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:38.609 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:38.609 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:38.609 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:38.610 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:38.610 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:38.610 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:38.610 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:38.610 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:38.610 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:38.611 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:38.611 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:38.611 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:38.611 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:38.611 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:38.611 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:38.611 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:38.611 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:38.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:38.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:38.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:38.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:38.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:38.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:38.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:38.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:38.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:39.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:39.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:39.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:39.165 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:39.659 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:39.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:39.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:39.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:39.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:39.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:39.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:39.660 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:39.661 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:40.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15725910 2020-03-11T22:51:40.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15725910 '!=' null 2020-03-11T22:51:40.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:40.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:40.145 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:40.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:40.236 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:40.237 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:40.237 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:40.237 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:40.237 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:40.237 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:40.237 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:40.238 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:40.238 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:40.238 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:40.238 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:40.238 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:40.238 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:40.238 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:40.238 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:40.240 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:40.240 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:40.240 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:40.240 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:40.240 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:40.240 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:40.240 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:40.241 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:40.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:40.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:40.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:40.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:40.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:40.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:40.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:40.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:40.347 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:40.705 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:40.705 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:40.707 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:40.707 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:41.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:41.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:41.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:41.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:41.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:41.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:41.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:41.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:41.335 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:41.835 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28830527 2020-03-11T22:51:41.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28830527 '!=' null 2020-03-11T22:51:41.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:41.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:41.836 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:41.836 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:41.929 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:41.929 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:41.929 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:41.929 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:41.930 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:41.930 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:41.930 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:41.930 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:41.930 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:41.930 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:41.930 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:41.932 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:41.932 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:41.932 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:41.932 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:41.932 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:41.932 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:41.932 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:41.932 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:41.933 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:41.933 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:41.933 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:41.933 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:41.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:42.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:42.037 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:42.038 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:42.038 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:42.038 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:42.039 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:42.039 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:42.039 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:42.399 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:42.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:42.400 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:42.400 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:42.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:42.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:51:42.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:42.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:51:42.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:42.937 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:42.938 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:43.028 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:43.028 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:43.028 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:43.029 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:43.029 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:43.029 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:43.029 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:43.029 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:43.029 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:43.030 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:43.030 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:43.030 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:43.030 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:43.030 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:43.030 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:43.031 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:43.031 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:43.031 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:43.031 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:43.031 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:43.031 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:43.032 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:43.032 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:43.032 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:43.032 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:43.032 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:43.032 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:43.032 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:43.032 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:43.033 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:43.033 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:43.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:43.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:43.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:43.138 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:43.138 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:43.139 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:43.139 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:43.139 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:43.139 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:43.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:43.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:43.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:43.498 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:43.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:43.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:43.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:43.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:43.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:43.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:43.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:43.975 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:43.976 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:44.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28830551 2020-03-11T22:51:44.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28830551 '!=' null 2020-03-11T22:51:44.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:44.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:44.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:44.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:44.609 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:44.609 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:44.609 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:44.609 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:44.609 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:44.610 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:44.610 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:44.610 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:44.610 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:44.610 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:44.610 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:44.610 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:44.611 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:44.611 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:44.611 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:44.611 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:44.611 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:44.611 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:44.611 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:44.612 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:44.612 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:44.612 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:44.612 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:44.612 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:44.612 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:44.612 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:44.612 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:44.613 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:44.613 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:44.613 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:44.613 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:44.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:44.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:44.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:44.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:44.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:44.719 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:44.719 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:44.719 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:44.719 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:45.216 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:45.216 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:45.216 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:45.217 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:45.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:45.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:45.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:45.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:45.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:45.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:45.703 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:45.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:45.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:46.175 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=12140801 2020-03-11T22:51:46.175 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 12140801 '!=' null 2020-03-11T22:51:46.175 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:46.176 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:46.176 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:46.176 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:46.269 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:46.269 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:46.269 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:46.269 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:46.270 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:46.270 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:46.270 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:46.270 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:46.270 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:46.270 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:46.270 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:46.271 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:46.271 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:46.271 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:46.271 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:46.271 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:46.271 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:46.271 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:46.272 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:46.272 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:46.272 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:46.272 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:46.272 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:46.272 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:46.272 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:46.273 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:46.273 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:46.273 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:46.273 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:46.273 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:46.273 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:46.277 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:46.378 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:46.379 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:46.379 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:46.379 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:46.379 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:46.379 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:46.379 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:46.380 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:46.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:46.734 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:46.735 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:46.736 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:47.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:47.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:47.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:47.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:47.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:47.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:47.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:47.244 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:47.245 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:47.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22353835 2020-03-11T22:51:47.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22353835 '!=' null 2020-03-11T22:51:47.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:47.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:47.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:47.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:47.809 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:47.809 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:47.810 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:47.810 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:47.810 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:47.810 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:47.810 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:47.810 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:47.810 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:47.811 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:47.811 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:47.811 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:47.811 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:47.811 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:47.811 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:47.811 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:47.812 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:47.812 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:47.812 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:47.812 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:47.812 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:47.812 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:47.813 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:47.813 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:47.813 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:47.813 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:47.813 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:47.813 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:47.813 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:47.814 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:47.814 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:47.816 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:47.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:47.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:47.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:47.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:47.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:47.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:47.919 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:47.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:48.277 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:48.277 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:48.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:48.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:48.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:48.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:48.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:48.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:48.759 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:48.759 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:49.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220538 2020-03-11T22:51:49.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220538 '!=' null 2020-03-11T22:51:49.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:49.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:49.314 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:49.314 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:49.405 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:49.406 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:49.406 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:49.406 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:49.406 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:49.406 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:49.406 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:49.406 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:49.407 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:49.407 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:49.407 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:49.407 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:49.407 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:49.407 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:49.407 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:49.408 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:49.408 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:49.408 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:49.408 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:49.408 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:49.408 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:49.408 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:49.409 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:49.409 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:49.409 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:49.409 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:49.409 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:49.409 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:49.409 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:49.409 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:49.410 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:49.413 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:49.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:49.516 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:49.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:49.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:49.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:49.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:49.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:49.517 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:49.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:49.881 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:49.881 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:49.882 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:50.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:50.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:50.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:50.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:50.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:50.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:50.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:50.360 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:50.360 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:50.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220538 2020-03-11T22:51:50.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220538 '!=' null 2020-03-11T22:51:50.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:50.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:50.899 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:50.899 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:50.993 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:50.993 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:50.993 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:50.993 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:50.994 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:50.994 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:50.994 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:50.994 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:50.994 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:50.994 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:50.994 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:50.995 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:50.995 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:50.995 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:50.995 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:50.995 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:50.995 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:50.995 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:50.996 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:50.996 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:50.996 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:50.996 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:50.996 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:50.996 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:50.996 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:50.996 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:50.997 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:50.997 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:50.997 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:50.997 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:50.997 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:51.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:51.102 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:51.102 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:51.105 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:51.105 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:51.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:51.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:51.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:51.106 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:51.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:51.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:51.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:51.458 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:51.935 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:51.935 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:51.935 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:51.935 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:51.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:51.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:51.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:51.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:51.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:52.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28830633 2020-03-11T22:51:52.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28830633 '!=' null 2020-03-11T22:51:52.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:52.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:52.424 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:52.425 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:52.516 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:52.516 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:52.517 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:52.517 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:52.517 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:52.517 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:52.517 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:52.517 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:52.518 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:52.518 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:52.518 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:52.518 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:52.518 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:52.518 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:52.518 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:52.519 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:52.519 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:52.519 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:52.519 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:52.519 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:52.519 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:52.519 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:52.520 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:52.520 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:52.520 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:52.520 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:52.520 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:52.520 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:52.521 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:52.521 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:52.521 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:52.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:52.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:52.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:52.626 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:52.626 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:52.626 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:52.626 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:52.627 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:52.627 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:52.982 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:52.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:52.983 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:52.983 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:53.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:53.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:53.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:53.463 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:53.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:53.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:53.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:53.464 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:53.464 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:54.021 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094202 2020-03-11T22:51:54.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094202 '!=' null 2020-03-11T22:51:54.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:54.022 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:54.022 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:54.022 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:54.113 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:54.113 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:54.114 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:54.114 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:54.114 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:54.114 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:54.114 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:54.114 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:54.114 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:54.115 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:54.115 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:54.115 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:54.115 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:54.115 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:54.115 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:54.115 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:54.116 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:54.116 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:54.116 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:54.116 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:54.116 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:54.116 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:54.116 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:54.116 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:54.117 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:54.117 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:54.117 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:54.117 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:54.117 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:54.117 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:54.117 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:54.120 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:54.222 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:54.222 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:54.223 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:54.223 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:54.223 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:54.223 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:54.223 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:54.223 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:54.585 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:54.585 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:54.585 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:54.585 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:55.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:55.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:55.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:55.065 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:55.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:55.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:55.066 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:55.066 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:55.066 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:55.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094218 2020-03-11T22:51:55.544 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094218 '!=' null 2020-03-11T22:51:55.544 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:55.544 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:55.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:55.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:55.635 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:55.636 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:55.636 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:55.636 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:55.636 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:55.636 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:55.636 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:55.637 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:55.637 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:55.637 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:55.637 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:55.637 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:55.637 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:55.637 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:55.638 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:55.638 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:55.638 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:55.638 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:55.638 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:55.638 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:55.638 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:55.639 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:55.639 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:55.639 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:55.639 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:55.639 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:55.639 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:55.639 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:55.640 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:55.640 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:55.640 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:55.643 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:55.744 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:55.744 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:55.745 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:55.745 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:55.745 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:55.745 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:55.746 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:55.746 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:56.103 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:56.103 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:56.104 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:56.104 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:56.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:56.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:56.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:56.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:56.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:56.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:56.589 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:56.589 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:56.589 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:57.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704506 2020-03-11T22:51:57.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704506 '!=' null 2020-03-11T22:51:57.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:57.173 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:57.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:57.174 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:57.266 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:57.266 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:57.266 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:57.266 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:57.266 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [ 2020-03-11T22:51:57.266 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:57.267 INFO:tasks.workunit.client.0.smithi016.stdout: "priority": 110, 2020-03-11T22:51:57.267 INFO:tasks.workunit.client.0.smithi016.stdout: "items": [ 2020-03-11T22:51:57.267 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:57.267 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:57.267 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:57.267 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:57.267 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:57.267 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:57.268 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:57.268 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:51:57.268 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:57.268 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:57.268 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "3.0", 2020-03-11T22:51:57.268 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 151, 2020-03-11T22:51:57.268 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:57.269 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:57.269 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:57.269 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:57.269 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:57.269 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:57.269 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:57.269 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:57.269 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:57.270 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:57.270 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:57.273 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:57.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:57.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:57.375 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:57.375 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:57.375 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:57.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:57.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:57.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:57.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:57.736 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:57.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:57.737 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:58.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:58.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:58.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:58.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:58.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:58.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:58.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:58.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:58.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:58.792 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704506 2020-03-11T22:51:58.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704506 '!=' null 2020-03-11T22:51:58.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:58.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:58.793 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:58.793 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:58.886 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:51:58.886 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:51:58.886 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:51:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:51:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:51:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:51:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:51:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:51:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:51:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:51:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:51:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:51:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:51:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:51:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:51:58.889 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:51:58.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:58.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:58.995 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:58.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:58.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:58.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:58.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:58.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:58.996 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:59.424 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:59.425 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:59.425 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:59.425 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:59.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:59.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:59.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:59.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:59.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:59.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:59.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:59.904 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:59.904 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:00.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23064418 2020-03-11T22:52:00.511 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23064418 '!=' null 2020-03-11T22:52:00.512 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:00.512 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:00.512 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:00.512 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:00.604 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:00.604 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:00.604 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:00.604 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:00.605 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:00.605 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:00.605 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:00.605 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:00.605 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:00.605 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:00.605 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:00.606 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:00.606 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:00.606 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:00.606 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:00.606 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:00.606 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:00.606 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:00.607 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:00.607 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:00.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:00.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:00.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:00.716 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:00.716 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:00.716 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:00.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:00.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:00.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:01.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:01.192 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:01.192 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:01.193 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:01.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:01.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 1 2020-03-11T22:52:01.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:52:01.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=2 2020-03-11T22:52:01.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:01.670 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:01.670 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:01.761 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:01.761 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:01.762 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:01.762 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:01.762 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:01.762 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:01.762 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:01.762 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:01.763 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:01.763 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:01.763 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:01.763 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:01.763 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:01.763 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:01.763 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:01.763 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:01.764 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:01.764 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:01.764 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:01.764 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:01.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:01.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:01.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:01.871 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:01.871 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:01.871 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:01.872 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:01.872 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:01.872 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:02.225 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:02.225 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:02.226 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:02.226 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:02.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:02.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:02.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:02.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:02.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:02.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:02.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:02.720 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:02.720 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:03.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831022 2020-03-11T22:52:03.249 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831022 '!=' null 2020-03-11T22:52:03.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:03.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:03.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:03.250 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:03.341 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:03.341 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:03.341 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:03.341 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:03.342 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:03.342 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:03.342 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:03.342 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:03.342 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:03.342 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:03.342 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:03.343 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:03.343 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:03.343 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:03.343 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:03.343 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:03.343 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:03.343 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:03.344 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:03.344 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:03.348 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:03.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:03.450 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:03.450 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:03.451 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:03.451 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:03.451 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:03.451 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:03.451 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:03.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:03.814 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:03.815 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:03.815 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:04.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:04.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:04.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:04.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:04.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:04.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:04.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:04.300 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:04.300 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:04.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831022 2020-03-11T22:52:04.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831022 '!=' null 2020-03-11T22:52:04.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:04.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:04.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:04.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:04.947 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:04.948 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:04.948 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:04.948 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:04.948 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:04.948 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:04.948 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:04.949 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:04.949 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:04.949 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:04.949 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:04.949 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:04.949 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:04.950 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:04.950 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:04.950 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:04.950 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:04.950 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:04.950 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:04.950 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:04.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:05.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:05.057 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:05.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:05.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:05.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:05.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:05.058 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:05.058 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:05.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:05.422 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:05.422 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:05.423 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:05.897 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:05.897 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:05.897 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:05.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:05.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:05.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:05.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:05.898 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:05.898 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415471 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415471 '!=' null 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:06.429 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:06.519 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:06.520 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:06.520 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:06.520 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:06.520 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:06.520 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:06.520 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:06.521 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:06.521 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:06.521 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:06.521 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:06.521 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:06.521 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:06.521 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:06.522 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:06.522 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:06.522 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:06.522 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:06.522 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:06.522 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:06.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:06.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:06.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:06.629 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:06.629 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:06.629 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:06.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:06.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:06.630 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:06.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:06.989 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:06.990 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:06.990 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:07.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:07.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:07.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:07.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:07.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:07.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:07.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:07.496 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:07.496 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:07.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28957045 2020-03-11T22:52:07.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28957045 '!=' null 2020-03-11T22:52:07.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:07.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:07.979 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:07.979 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:08.072 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:08.072 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:08.072 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:08.072 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:08.073 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:08.073 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:08.073 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:08.073 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:08.073 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:08.073 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:08.073 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:08.074 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:08.074 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:08.074 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:08.074 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:08.074 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:08.074 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:08.074 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:08.075 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:08.075 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:08.079 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:08.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:08.181 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:08.181 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:08.181 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:08.181 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:08.182 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:08.182 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:08.182 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:08.544 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:08.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:08.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:08.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:09.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:09.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:09.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:09.073 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:09.073 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:09.073 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:09.073 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:09.073 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:09.073 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:09.624 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24113319 2020-03-11T22:52:09.624 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24113319 '!=' null 2020-03-11T22:52:09.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:09.625 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:09.625 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:09.625 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:09.716 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:09.716 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:09.717 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:09.717 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:09.717 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:09.717 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:09.717 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:09.717 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:09.717 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:09.718 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:09.718 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:09.718 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:09.718 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:09.718 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:09.718 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:09.718 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:09.719 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:09.719 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:09.719 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:09.719 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:09.724 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:09.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:09.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:09.826 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:09.827 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:09.827 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:09.827 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:09.827 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:09.827 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:10.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:10.186 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:10.186 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:10.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:10.667 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:10.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:10.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:10.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:10.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:10.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:10.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:10.668 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:10.669 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:11.145 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30099621 2020-03-11T22:52:11.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30099621 '!=' null 2020-03-11T22:52:11.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:11.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:11.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:11.146 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:11.238 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:11.239 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:11.239 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:11.239 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:11.239 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:11.239 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:11.239 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:11.239 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:11.240 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:11.240 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:11.240 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:11.240 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:11.240 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:11.240 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:11.240 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:11.241 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:11.241 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:11.241 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:11.241 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:11.241 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:11.246 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:11.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:11.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:11.348 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:11.348 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:11.348 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:11.348 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:11.349 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:11.349 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:11.708 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:11.708 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:11.708 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:11.708 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:12.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:12.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:12.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:12.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:12.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:12.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:12.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:12.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:12.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30099621 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30099621 '!=' null 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:12.752 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:12.752 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:12.753 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:12.753 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:12.753 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:12.753 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:12.753 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:12.753 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:12.753 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:12.754 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:12.754 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:12.754 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:12.754 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:12.754 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:12.754 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:12.754 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:12.755 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:12.755 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:12.755 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:12.755 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:12.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:12.861 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:12.861 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:12.862 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:12.863 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:12.863 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:12.863 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:12.863 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:12.863 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:13.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:13.221 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:13.221 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:13.221 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:13.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:13.716 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:13.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:13.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:13.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:13.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:13.717 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:13.717 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:13.718 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:14.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094382 2020-03-11T22:52:14.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094382 '!=' null 2020-03-11T22:52:14.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:14.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:14.195 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:14.195 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:14.288 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:14.289 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:14.289 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:14.289 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:14.289 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:14.289 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:14.289 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:14.290 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:14.290 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:14.290 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:14.290 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:14.290 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:14.290 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:14.291 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:14.291 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:14.291 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:14.291 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:14.291 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:14.291 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:14.291 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:14.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:14.398 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:14.398 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:14.398 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:14.398 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:14.398 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:14.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:14.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:14.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:14.756 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:14.756 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:14.757 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:14.757 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:15.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:15.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:15.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:15.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:15.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:15.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:15.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:15.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:15.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:15.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094371 2020-03-11T22:52:15.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094371 '!=' null 2020-03-11T22:52:15.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:15.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:15.822 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:15.823 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:15.914 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:15.914 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:15.914 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:15.915 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:15.915 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:15.915 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:15.915 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:15.915 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:15.915 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:15.915 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:15.916 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:15.916 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:15.916 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:15.916 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:15.916 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:15.916 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:15.916 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:15.917 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:15.917 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:15.917 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:15.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:16.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:16.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:16.384 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:16.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:16.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:16.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:16.938 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:16.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:16.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:16.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:16.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:16.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:16.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:16.939 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:16.940 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:17.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578400 2020-03-11T22:52:17.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578400 '!=' null 2020-03-11T22:52:17.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:17.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:17.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:17.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:17.686 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:17.687 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:17.687 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:17.687 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:17.687 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:17.687 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:17.687 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:17.688 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:17.688 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:17.688 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:17.688 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:17.688 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:17.688 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:17.688 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:17.688 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:17.689 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:17.689 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:17.689 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:17.689 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:17.689 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:17.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:17.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:17.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:17.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:17.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:17.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:17.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:17.797 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:17.797 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:18.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:18.154 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:18.154 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:18.154 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:18.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:18.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:18.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:18.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:18.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:18.645 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:18.645 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:18.645 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:18.645 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:19.174 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094487 2020-03-11T22:52:19.174 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094487 '!=' null 2020-03-11T22:52:19.174 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:19.175 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:19.175 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:19.175 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:19.267 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:19.267 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:19.267 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:19.267 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:19.267 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:19.267 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [ 2020-03-11T22:52:19.268 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:19.268 INFO:tasks.workunit.client.0.smithi016.stdout: "item": "1.0", 2020-03-11T22:52:19.268 INFO:tasks.workunit.client.0.smithi016.stdout: "prio": 110, 2020-03-11T22:52:19.268 INFO:tasks.workunit.client.0.smithi016.stdout: "can_preempt": true 2020-03-11T22:52:19.268 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:19.268 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:19.268 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:19.269 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:19.269 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:19.269 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:19.269 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:19.269 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:19.269 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:19.269 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:19.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:19.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:19.377 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:19.740 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:19.741 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:19.741 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:19.742 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:20.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:20.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:20.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:20.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:20.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:20.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:20.219 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:20.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:20.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:20.764 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094487 2020-03-11T22:52:20.764 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094487 '!=' null 2020-03-11T22:52:20.764 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:20.765 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:20.857 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:20.857 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:20.857 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:20.857 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:20.858 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:20.858 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:20.858 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:20.858 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:20.858 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:20.858 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:20.858 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:20.859 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:20.859 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:20.859 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:20.864 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:20.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:20.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:20.966 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:20.967 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:20.967 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:20.967 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:20.967 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:20.967 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:21.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:21.322 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:21.322 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:21.322 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:21.794 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:21.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:21.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:21.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:21.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:21.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:21.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:21.795 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:21.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:22.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30141647 2020-03-11T22:52:22.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30141647 '!=' null 2020-03-11T22:52:22.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:22.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:22.296 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:22.297 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:22.387 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:22.388 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:22.388 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:22.388 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:22.388 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:22.388 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:22.388 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:22.388 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:22.389 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:22.389 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:22.389 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:22.389 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:22.389 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:22.389 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:22.395 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:22.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:22.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:22.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:22.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:22.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:22.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:22.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:22.497 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:22.861 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:22.862 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:22.862 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:22.862 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:23.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:23.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:23.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:23.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:23.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:23.338 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:23.339 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:23.339 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:23.339 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:23.840 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30141796 2020-03-11T22:52:23.840 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30141796 '!=' null 2020-03-11T22:52:23.840 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:23.841 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:23.841 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:23.841 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:23.930 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:23.931 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:23.931 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:23.931 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:23.931 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:23.931 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:23.931 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:23.931 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:23.932 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:23.932 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:23.932 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:23.932 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:23.932 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:23.932 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:23.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:24.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:24.039 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:24.040 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:24.040 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:24.040 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:24.040 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:24.040 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:24.040 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:24.395 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:24.395 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:24.395 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:24.395 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:24.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:24.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:24.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:24.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:24.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:24.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:24.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:24.875 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:24.876 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:25.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415695 2020-03-11T22:52:25.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415695 '!=' null 2020-03-11T22:52:25.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:25.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:25.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:25.346 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:25.437 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:25.437 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:25.438 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:25.438 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:25.438 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:25.438 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:25.438 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:25.438 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:25.438 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:25.438 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:25.439 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:25.439 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:25.439 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:25.439 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:25.444 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:25.546 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:25.546 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:25.547 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:25.547 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:25.547 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:25.547 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:25.547 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:25.547 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:25.902 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:25.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:25.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:25.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:26.393 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:26.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:26.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:26.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:26.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:26.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:26.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:26.395 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:26.395 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:26.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415722 2020-03-11T22:52:26.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415722 '!=' null 2020-03-11T22:52:26.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:26.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:26.872 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:26.872 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:26.963 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:26.963 INFO:tasks.workunit.client.0.smithi016.stdout: "local_reservations": { 2020-03-11T22:52:26.963 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:26.964 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:26.964 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:26.964 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:26.964 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:26.964 INFO:tasks.workunit.client.0.smithi016.stdout: "remote_reservations": { 2020-03-11T22:52:26.964 INFO:tasks.workunit.client.0.smithi016.stdout: "max_allowed": 1, 2020-03-11T22:52:26.964 INFO:tasks.workunit.client.0.smithi016.stdout: "min_priority": 0, 2020-03-11T22:52:26.965 INFO:tasks.workunit.client.0.smithi016.stdout: "queues": [], 2020-03-11T22:52:26.965 INFO:tasks.workunit.client.0.smithi016.stdout: "in_progress": [] 2020-03-11T22:52:26.965 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:26.965 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:26.970 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:27.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:27.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:27.072 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:27.072 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:27.073 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:27.073 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:27.073 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:27.073 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:27.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:52:27.426 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:27.426 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:27.427 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:27.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:52:27.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:52:27.915 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:52:27.915 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:52:28.264 INFO:tasks.workunit.client.0.smithi016.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:52:28.265 INFO:tasks.workunit.client.0.smithi016.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:51:36.481179+0000 93'150 118:340 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:48:54.370142+0000 0'0 2020-03-11T22:48:54.370142+0000 0 2020-03-11T22:52:28.265 INFO:tasks.workunit.client.0.smithi016.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:20.875619+0000 93'150 119:354 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:40.040132+0000 0'0 2020-03-11T22:48:40.040132+0000 0 2020-03-11T22:52:28.265 INFO:tasks.workunit.client.0.smithi016.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:51:59.061849+0000 93'150 118:356 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:44.827101+0000 0'0 2020-03-11T22:48:44.827101+0000 0 2020-03-11T22:52:28.265 INFO:tasks.workunit.client.0.smithi016.stdout: 2020-03-11T22:52:28.265 INFO:tasks.workunit.client.0.smithi016.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:52:28.266 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:52:28.275 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:28.275 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:28.276 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:28.276 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:28.276 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:28.276 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:52:28.276 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56083/ceph-osd.1.asok 2020-03-11T22:52:28.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:52:28.277 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56083/ceph-osd.1.asok dump_pgstate_history 2020-03-11T22:52:28.369 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:52:28.370 INFO:tasks.workunit.client.0.smithi016.stdout: "pgs": [ 2020-03-11T22:52:28.370 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.370 INFO:tasks.workunit.client.0.smithi016.stdout: "pg": "7.0", 2020-03-11T22:52:28.370 INFO:tasks.workunit.client.0.smithi016.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.370 INFO:tasks.workunit.client.0.smithi016.stdout: "history": [ 2020-03-11T22:52:28.370 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.371 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "62", 2020-03-11T22:52:28.371 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.371 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.371 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.371 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.796591+0000", 2020-03-11T22:52:28.371 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:54.796600+0000" 2020-03-11T22:52:28.371 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.372 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.372 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.372 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.796606+0000", 2020-03-11T22:52:28.372 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:54.797106+0000" 2020-03-11T22:52:28.372 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.372 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.373 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.373 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.797106+0000", 2020-03-11T22:52:28.373 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:54.797173+0000" 2020-03-11T22:52:28.373 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.373 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.373 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.373 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.797173+0000", 2020-03-11T22:52:28.374 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:54.797186+0000" 2020-03-11T22:52:28.374 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.374 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.374 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.374 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.797187+0000", 2020-03-11T22:52:28.374 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:54.965717+0000" 2020-03-11T22:52:28.374 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.374 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.375 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.375 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.796603+0000", 2020-03-11T22:52:28.375 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:54.965731+0000" 2020-03-11T22:52:28.375 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.375 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.375 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.375 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.965935+0000", 2020-03-11T22:52:28.375 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:55.922837+0000" 2020-03-11T22:52:28.376 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.376 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.376 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.376 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:55.922838+0000", 2020-03-11T22:52:28.376 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:55.922936+0000" 2020-03-11T22:52:28.376 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.376 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.377 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.377 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:55.922936+0000", 2020-03-11T22:52:28.377 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.499683+0000" 2020-03-11T22:52:28.377 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.377 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.377 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.377 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.965731+0000", 2020-03-11T22:52:28.378 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.499701+0000" 2020-03-11T22:52:28.378 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.378 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.378 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.378 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.796601+0000", 2020-03-11T22:52:28.378 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.499715+0000" 2020-03-11T22:52:28.379 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.379 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.379 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.379 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:54.796588+0000", 2020-03-11T22:52:28.379 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.499719+0000" 2020-03-11T22:52:28.379 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.379 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.380 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.380 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.380 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "63", 2020-03-11T22:52:28.380 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.380 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.380 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.380 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.499719+0000", 2020-03-11T22:52:28.381 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.499935+0000" 2020-03-11T22:52:28.381 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.381 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.381 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.381 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.381 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "91", 2020-03-11T22:52:28.381 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.382 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.382 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.382 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.499945+0000", 2020-03-11T22:52:28.382 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.499956+0000" 2020-03-11T22:52:28.382 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.382 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.382 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.383 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.499963+0000", 2020-03-11T22:52:28.383 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.662104+0000" 2020-03-11T22:52:28.383 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.383 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.383 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.383 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.662105+0000", 2020-03-11T22:52:28.383 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.662169+0000" 2020-03-11T22:52:28.383 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.384 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.384 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.384 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.662169+0000", 2020-03-11T22:52:28.384 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.662183+0000" 2020-03-11T22:52:28.384 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.384 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.384 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.385 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.662183+0000", 2020-03-11T22:52:28.385 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.757478+0000" 2020-03-11T22:52:28.385 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.385 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.385 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.385 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.499960+0000", 2020-03-11T22:52:28.385 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.757489+0000" 2020-03-11T22:52:28.386 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.386 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.386 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.386 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.757613+0000", 2020-03-11T22:52:28.386 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.926930+0000" 2020-03-11T22:52:28.386 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.386 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.386 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.387 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.926930+0000", 2020-03-11T22:52:28.387 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:56.927028+0000" 2020-03-11T22:52:28.387 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.387 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.387 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.387 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.927029+0000", 2020-03-11T22:52:28.387 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.275528+0000" 2020-03-11T22:52:28.387 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.388 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.388 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.388 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.757489+0000", 2020-03-11T22:52:28.388 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.275546+0000" 2020-03-11T22:52:28.388 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.388 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.388 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.389 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.499957+0000", 2020-03-11T22:52:28.389 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.275560+0000" 2020-03-11T22:52:28.389 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.389 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.389 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.389 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:56.499935+0000", 2020-03-11T22:52:28.389 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.275564+0000" 2020-03-11T22:52:28.389 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.390 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.390 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.390 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.390 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "91", 2020-03-11T22:52:28.390 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.390 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.390 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.391 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.275565+0000", 2020-03-11T22:52:28.391 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.275735+0000" 2020-03-11T22:52:28.391 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.391 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.391 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.391 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.391 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "98", 2020-03-11T22:52:28.391 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.392 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.392 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.392 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.275738+0000", 2020-03-11T22:52:28.392 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.275748+0000" 2020-03-11T22:52:28.392 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.392 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.392 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.392 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.275755+0000", 2020-03-11T22:52:28.393 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.303775+0000" 2020-03-11T22:52:28.393 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.393 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.393 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.393 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.303776+0000", 2020-03-11T22:52:28.393 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.303829+0000" 2020-03-11T22:52:28.393 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.393 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.394 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.394 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.303829+0000", 2020-03-11T22:52:28.394 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.303842+0000" 2020-03-11T22:52:28.394 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.394 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.394 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.394 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.303843+0000", 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.593463+0000" 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.275752+0000", 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.593476+0000" 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.396 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.396 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.593547+0000", 2020-03-11T22:52:28.396 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.618674+0000" 2020-03-11T22:52:28.396 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.396 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.396 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.396 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.618675+0000", 2020-03-11T22:52:28.397 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.618725+0000" 2020-03-11T22:52:28.397 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.397 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.397 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.397 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.618725+0000", 2020-03-11T22:52:28.397 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.848740+0000" 2020-03-11T22:52:28.397 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.398 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.398 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.398 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.593476+0000", 2020-03-11T22:52:28.398 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.848766+0000" 2020-03-11T22:52:28.398 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.398 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.398 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.399 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.275749+0000", 2020-03-11T22:52:28.399 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.848819+0000" 2020-03-11T22:52:28.399 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.399 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.399 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.399 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.275735+0000", 2020-03-11T22:52:28.399 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.848824+0000" 2020-03-11T22:52:28.399 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.400 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.400 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.400 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.400 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "99", 2020-03-11T22:52:28.400 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.400 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.400 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.401 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.848825+0000", 2020-03-11T22:52:28.401 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.849207+0000" 2020-03-11T22:52:28.401 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.401 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.401 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.401 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.401 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "100", 2020-03-11T22:52:28.401 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.402 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.402 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.402 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.849212+0000", 2020-03-11T22:52:28.402 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.849230+0000" 2020-03-11T22:52:28.402 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.402 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.402 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.402 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.849238+0000", 2020-03-11T22:52:28.403 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.849788+0000" 2020-03-11T22:52:28.403 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.403 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.403 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.403 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.849789+0000", 2020-03-11T22:52:28.403 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.849853+0000" 2020-03-11T22:52:28.403 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.403 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.404 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.404 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.849235+0000", 2020-03-11T22:52:28.404 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.849862+0000" 2020-03-11T22:52:28.404 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.404 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.404 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:28.404 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.849862+0000", 2020-03-11T22:52:28.404 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.995510+0000" 2020-03-11T22:52:28.405 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.405 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.405 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.405 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.849231+0000", 2020-03-11T22:52:28.405 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.995528+0000" 2020-03-11T22:52:28.405 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.405 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.406 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.406 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.849208+0000", 2020-03-11T22:52:28.406 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.995533+0000" 2020-03-11T22:52:28.406 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.406 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.406 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.406 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.406 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "100", 2020-03-11T22:52:28.407 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.407 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.407 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.407 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.995533+0000", 2020-03-11T22:52:28.407 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.995764+0000" 2020-03-11T22:52:28.407 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.407 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.408 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.408 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.408 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "114", 2020-03-11T22:52:28.408 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.408 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.408 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.408 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.995769+0000", 2020-03-11T22:52:28.408 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:25.995784+0000" 2020-03-11T22:52:28.409 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.409 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.409 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.409 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.995795+0000", 2020-03-11T22:52:28.409 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:26.014765+0000" 2020-03-11T22:52:28.409 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.409 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.410 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.410 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:26.014765+0000", 2020-03-11T22:52:28.410 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:26.014849+0000" 2020-03-11T22:52:28.410 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.410 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.410 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.410 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:26.014849+0000", 2020-03-11T22:52:28.410 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:26.014869+0000" 2020-03-11T22:52:28.411 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.411 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.411 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.411 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:26.014870+0000", 2020-03-11T22:52:28.411 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:26.195650+0000" 2020-03-11T22:52:28.411 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.411 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.412 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.412 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.995791+0000", 2020-03-11T22:52:28.412 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:26.195660+0000" 2020-03-11T22:52:28.412 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.412 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.412 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.412 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:26.196046+0000", 2020-03-11T22:52:28.412 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:26.598376+0000" 2020-03-11T22:52:28.413 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.413 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.413 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:28.413 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:26.598377+0000", 2020-03-11T22:52:28.413 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:26.598540+0000" 2020-03-11T22:52:28.414 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.414 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.414 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:28.414 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:26.598540+0000", 2020-03-11T22:52:28.414 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:26.599009+0000" 2020-03-11T22:52:28.414 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.414 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.414 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:28.415 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:26.599010+0000", 2020-03-11T22:52:28.415 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:58.535074+0000" 2020-03-11T22:52:28.415 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.415 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.415 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:52:28.415 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:58.535074+0000", 2020-03-11T22:52:28.416 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:58.535300+0000" 2020-03-11T22:52:28.416 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.416 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.416 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:28.416 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:58.535300+0000", 2020-03-11T22:52:28.416 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:10.746958+0000" 2020-03-11T22:52:28.416 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.417 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.417 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:28.417 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:10.746958+0000", 2020-03-11T22:52:28.417 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:10.747498+0000" 2020-03-11T22:52:28.417 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.417 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.417 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:10.747498+0000", 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:35.972879+0000" 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:35.972880+0000", 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:35.973054+0000" 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.419 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.419 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.419 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:35.973054+0000", 2020-03-11T22:52:28.419 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:36.087332+0000" 2020-03-11T22:52:28.419 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.419 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.419 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.420 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:26.195661+0000", 2020-03-11T22:52:28.420 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:36.087386+0000" 2020-03-11T22:52:28.420 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.420 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.420 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.420 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.995785+0000", 2020-03-11T22:52:28.420 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:36.087426+0000" 2020-03-11T22:52:28.420 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.421 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.421 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.421 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:25.995764+0000", 2020-03-11T22:52:28.421 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:36.087438+0000" 2020-03-11T22:52:28.421 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.421 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.421 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.422 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.422 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "114", 2020-03-11T22:52:28.422 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.422 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.422 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.422 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:36.087439+0000", 2020-03-11T22:52:28.422 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:36.087957+0000" 2020-03-11T22:52:28.423 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.423 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.423 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.423 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.423 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.423 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.423 INFO:tasks.workunit.client.0.smithi016.stdout: "pg": "3.0", 2020-03-11T22:52:28.423 INFO:tasks.workunit.client.0.smithi016.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.424 INFO:tasks.workunit.client.0.smithi016.stdout: "history": [ 2020-03-11T22:52:28.424 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.424 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "40", 2020-03-11T22:52:28.424 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.424 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.424 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.424 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.530198+0000", 2020-03-11T22:52:28.424 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:45.530208+0000" 2020-03-11T22:52:28.425 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.425 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.425 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.425 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.530214+0000", 2020-03-11T22:52:28.425 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:45.714177+0000" 2020-03-11T22:52:28.425 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.425 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.426 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.426 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.714177+0000", 2020-03-11T22:52:28.426 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:45.714237+0000" 2020-03-11T22:52:28.426 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.426 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.426 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.426 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.714238+0000", 2020-03-11T22:52:28.426 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:45.714250+0000" 2020-03-11T22:52:28.427 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.427 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.427 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.427 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.714250+0000", 2020-03-11T22:52:28.427 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:45.798434+0000" 2020-03-11T22:52:28.427 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.427 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.428 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.428 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.530211+0000", 2020-03-11T22:52:28.428 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:45.798451+0000" 2020-03-11T22:52:28.428 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.428 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.428 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.429 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.798737+0000", 2020-03-11T22:52:28.429 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.703981+0000" 2020-03-11T22:52:28.429 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.429 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.429 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.429 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.703982+0000", 2020-03-11T22:52:28.429 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.704073+0000" 2020-03-11T22:52:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.704073+0000", 2020-03-11T22:52:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.916664+0000" 2020-03-11T22:52:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.430 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.798451+0000", 2020-03-11T22:52:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.916699+0000" 2020-03-11T22:52:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.530208+0000", 2020-03-11T22:52:28.431 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.916725+0000" 2020-03-11T22:52:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:45.530195+0000", 2020-03-11T22:52:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.916735+0000" 2020-03-11T22:52:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.432 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "40", 2020-03-11T22:52:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.916735+0000", 2020-03-11T22:52:28.433 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.917082+0000" 2020-03-11T22:52:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "89", 2020-03-11T22:52:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.434 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.917090+0000", 2020-03-11T22:52:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.917114+0000" 2020-03-11T22:52:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.917130+0000", 2020-03-11T22:52:28.435 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.938839+0000" 2020-03-11T22:52:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.938840+0000", 2020-03-11T22:52:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.938894+0000" 2020-03-11T22:52:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.436 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.938895+0000", 2020-03-11T22:52:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:46.938906+0000" 2020-03-11T22:52:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.938906+0000", 2020-03-11T22:52:28.437 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:47.331667+0000" 2020-03-11T22:52:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.917120+0000", 2020-03-11T22:52:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:47.331684+0000" 2020-03-11T22:52:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.438 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:47.331905+0000", 2020-03-11T22:52:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:47.648779+0000" 2020-03-11T22:52:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.439 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:47.648780+0000", 2020-03-11T22:52:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:47.648827+0000" 2020-03-11T22:52:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.440 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:47.648828+0000", 2020-03-11T22:52:28.441 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.082596+0000" 2020-03-11T22:52:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.442 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:47.331684+0000", 2020-03-11T22:52:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.082612+0000" 2020-03-11T22:52:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.443 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.917114+0000", 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.082625+0000" 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:46.917083+0000", 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.082629+0000" 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "89", 2020-03-11T22:52:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.445 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.082629+0000", 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.082830+0000" 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "106", 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.082834+0000", 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.082845+0000" 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.082851+0000", 2020-03-11T22:52:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.136741+0000" 2020-03-11T22:52:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.136742+0000", 2020-03-11T22:52:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.136794+0000" 2020-03-11T22:52:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.448 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.136794+0000", 2020-03-11T22:52:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.136808+0000" 2020-03-11T22:52:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.136808+0000", 2020-03-11T22:52:28.449 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.276035+0000" 2020-03-11T22:52:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.082848+0000", 2020-03-11T22:52:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.276048+0000" 2020-03-11T22:52:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.450 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.276166+0000", 2020-03-11T22:52:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.364365+0000" 2020-03-11T22:52:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.364366+0000", 2020-03-11T22:52:28.451 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:49:22.364453+0000" 2020-03-11T22:52:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.364454+0000", 2020-03-11T22:52:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.918025+0000" 2020-03-11T22:52:28.452 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.276048+0000", 2020-03-11T22:52:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.918072+0000" 2020-03-11T22:52:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.453 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.082845+0000", 2020-03-11T22:52:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.918133+0000" 2020-03-11T22:52:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:49:22.082831+0000", 2020-03-11T22:52:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.918141+0000" 2020-03-11T22:52:28.454 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "106", 2020-03-11T22:52:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.455 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:44.918142+0000", 2020-03-11T22:52:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.918633+0000" 2020-03-11T22:52:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "108", 2020-03-11T22:52:28.456 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:44.918645+0000", 2020-03-11T22:52:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.918673+0000" 2020-03-11T22:52:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.457 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:44.918690+0000", 2020-03-11T22:52:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.919307+0000" 2020-03-11T22:52:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:44.919308+0000", 2020-03-11T22:52:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.919415+0000" 2020-03-11T22:52:28.458 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:44.918683+0000", 2020-03-11T22:52:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:44.919437+0000" 2020-03-11T22:52:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:28.459 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:44.919438+0000", 2020-03-11T22:52:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.093957+0000" 2020-03-11T22:52:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:44.918674+0000", 2020-03-11T22:52:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.093975+0000" 2020-03-11T22:52:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.460 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:44.918633+0000", 2020-03-11T22:52:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.093980+0000" 2020-03-11T22:52:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.461 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "108", 2020-03-11T22:52:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.093980+0000", 2020-03-11T22:52:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.094198+0000" 2020-03-11T22:52:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.462 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "116", 2020-03-11T22:52:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.094203+0000", 2020-03-11T22:52:28.463 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.094219+0000" 2020-03-11T22:52:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.094229+0000", 2020-03-11T22:52:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.111156+0000" 2020-03-11T22:52:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.464 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.111156+0000", 2020-03-11T22:52:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.111234+0000" 2020-03-11T22:52:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.465 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.111235+0000", 2020-03-11T22:52:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.111254+0000" 2020-03-11T22:52:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.111255+0000", 2020-03-11T22:52:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.378084+0000" 2020-03-11T22:52:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.466 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.094225+0000", 2020-03-11T22:52:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.378098+0000" 2020-03-11T22:52:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.467 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.378370+0000", 2020-03-11T22:52:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:45.419806+0000" 2020-03-11T22:52:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.419807+0000", 2020-03-11T22:52:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:58.535177+0000" 2020-03-11T22:52:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.468 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:58.535178+0000", 2020-03-11T22:52:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:58.535632+0000" 2020-03-11T22:52:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:28.469 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:58.535633+0000", 2020-03-11T22:52:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:10.746796+0000" 2020-03-11T22:52:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:52:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:10.746797+0000", 2020-03-11T22:52:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:10.747158+0000" 2020-03-11T22:52:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.470 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:10.747158+0000", 2020-03-11T22:52:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:35.973251+0000" 2020-03-11T22:52:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:35.973252+0000", 2020-03-11T22:52:28.471 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:35.973737+0000" 2020-03-11T22:52:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:35.973737+0000", 2020-03-11T22:52:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.260278+0000" 2020-03-11T22:52:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.472 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:58.260278+0000", 2020-03-11T22:52:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.260454+0000" 2020-03-11T22:52:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:58.260454+0000", 2020-03-11T22:52:28.473 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.392892+0000" 2020-03-11T22:52:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.378099+0000", 2020-03-11T22:52:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.392921+0000" 2020-03-11T22:52:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.474 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.094219+0000", 2020-03-11T22:52:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.392940+0000" 2020-03-11T22:52:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:45.094198+0000", 2020-03-11T22:52:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.392947+0000" 2020-03-11T22:52:28.475 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "116", 2020-03-11T22:52:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.476 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:58.392948+0000", 2020-03-11T22:52:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.393227+0000" 2020-03-11T22:52:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.477 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "pg": "1.0", 2020-03-11T22:52:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "history": [ 2020-03-11T22:52:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "26", 2020-03-11T22:52:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.478 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Initial", 2020-03-11T22:52:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.397619+0000", 2020-03-11T22:52:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.397736+0000" 2020-03-11T22:52:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "26", 2020-03-11T22:52:28.479 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.397736+0000", 2020-03-11T22:52:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.397769+0000" 2020-03-11T22:52:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.480 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "29", 2020-03-11T22:52:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.397772+0000", 2020-03-11T22:52:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.397783+0000" 2020-03-11T22:52:28.481 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.397790+0000", 2020-03-11T22:52:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.449504+0000" 2020-03-11T22:52:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.482 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.449505+0000", 2020-03-11T22:52:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.449594+0000" 2020-03-11T22:52:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.449595+0000", 2020-03-11T22:52:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.449608+0000" 2020-03-11T22:52:28.483 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.449609+0000", 2020-03-11T22:52:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.600622+0000" 2020-03-11T22:52:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.484 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.397787+0000", 2020-03-11T22:52:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.600630+0000" 2020-03-11T22:52:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.600827+0000", 2020-03-11T22:52:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.738713+0000" 2020-03-11T22:52:28.485 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.738714+0000", 2020-03-11T22:52:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:40.738760+0000" 2020-03-11T22:52:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.486 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.738760+0000", 2020-03-11T22:52:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.747887+0000" 2020-03-11T22:52:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.600631+0000", 2020-03-11T22:52:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.747908+0000" 2020-03-11T22:52:28.487 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.397783+0000", 2020-03-11T22:52:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.747924+0000" 2020-03-11T22:52:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.488 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:40.397769+0000", 2020-03-11T22:52:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.747928+0000" 2020-03-11T22:52:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "29", 2020-03-11T22:52:28.489 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.747928+0000", 2020-03-11T22:52:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.748116+0000" 2020-03-11T22:52:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.490 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "102", 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.748120+0000", 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.748130+0000" 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.748137+0000", 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.780325+0000" 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.780326+0000", 2020-03-11T22:52:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.780398+0000" 2020-03-11T22:52:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.780399+0000", 2020-03-11T22:52:28.493 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.780411+0000" 2020-03-11T22:52:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.780412+0000", 2020-03-11T22:52:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.962464+0000" 2020-03-11T22:52:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.494 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.748134+0000", 2020-03-11T22:52:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:41.962472+0000" 2020-03-11T22:52:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.962614+0000", 2020-03-11T22:52:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:42.015586+0000" 2020-03-11T22:52:28.495 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:42.015586+0000", 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:48:42.015632+0000" 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:42.015632+0000", 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.857381+0000" 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.962473+0000", 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.857403+0000" 2020-03-11T22:52:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.748131+0000", 2020-03-11T22:52:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.857441+0000" 2020-03-11T22:52:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.498 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:48:41.748116+0000", 2020-03-11T22:52:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.857445+0000" 2020-03-11T22:52:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.499 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "102", 2020-03-11T22:52:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:36.857446+0000", 2020-03-11T22:52:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.857699+0000" 2020-03-11T22:52:28.500 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "104", 2020-03-11T22:52:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.501 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:36.857703+0000", 2020-03-11T22:52:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.857715+0000" 2020-03-11T22:52:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:36.857723+0000", 2020-03-11T22:52:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.974819+0000" 2020-03-11T22:52:28.502 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:36.974820+0000", 2020-03-11T22:52:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.974943+0000" 2020-03-11T22:52:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.503 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:36.857720+0000", 2020-03-11T22:52:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:36.974963+0000" 2020-03-11T22:52:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:36.974964+0000", 2020-03-11T22:52:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.359578+0000" 2020-03-11T22:52:28.504 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:36.857716+0000", 2020-03-11T22:52:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.359597+0000" 2020-03-11T22:52:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.505 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:36.857700+0000", 2020-03-11T22:52:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.359603+0000" 2020-03-11T22:52:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.506 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "104", 2020-03-11T22:52:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.359603+0000", 2020-03-11T22:52:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.359829+0000" 2020-03-11T22:52:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.507 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "118", 2020-03-11T22:52:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Start", 2020-03-11T22:52:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.359833+0000", 2020-03-11T22:52:28.508 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.359849+0000" 2020-03-11T22:52:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.359857+0000", 2020-03-11T22:52:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.478109+0000" 2020-03-11T22:52:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.509 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.478110+0000", 2020-03-11T22:52:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.478206+0000" 2020-03-11T22:52:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.478206+0000", 2020-03-11T22:52:28.510 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.478231+0000" 2020-03-11T22:52:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.478232+0000", 2020-03-11T22:52:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.694377+0000" 2020-03-11T22:52:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.511 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.359853+0000", 2020-03-11T22:52:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.694392+0000" 2020-03-11T22:52:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.694700+0000", 2020-03-11T22:52:28.512 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:50:37.962011+0000" 2020-03-11T22:52:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.962012+0000", 2020-03-11T22:52:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.260664+0000" 2020-03-11T22:52:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.513 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:58.260665+0000", 2020-03-11T22:52:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:51:58.261144+0000" 2020-03-11T22:52:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.514 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:51:58.261145+0000", 2020-03-11T22:52:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:52:20.325893+0000" 2020-03-11T22:52:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:28.515 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:52:20.325894+0000", 2020-03-11T22:52:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:52:20.326098+0000" 2020-03-11T22:52:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:52:20.326099+0000", 2020-03-11T22:52:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:52:20.541375+0000" 2020-03-11T22:52:28.516 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.694392+0000", 2020-03-11T22:52:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:52:20.541396+0000" 2020-03-11T22:52:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started/Primary", 2020-03-11T22:52:28.517 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.359849+0000", 2020-03-11T22:52:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:52:20.541414+0000" 2020-03-11T22:52:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Started", 2020-03-11T22:52:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:50:37.359829+0000", 2020-03-11T22:52:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:52:20.541420+0000" 2020-03-11T22:52:28.518 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:52:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "epoch": "118", 2020-03-11T22:52:28.519 INFO:tasks.workunit.client.0.smithi016.stdout: "states": [ 2020-03-11T22:52:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:52:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: "state": "Reset", 2020-03-11T22:52:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: "enter": "2020-03-11T22:52:20.541421+0000", 2020-03-11T22:52:28.554 INFO:tasks.workunit.client.0.smithi016.stdout: "exit": "2020-03-11T22:52:20.541671+0000" 2020-03-11T22:52:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.555 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:52:28.556 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:52:28.556 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:52:28.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:52:28.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-03-11T22:52:28.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-03-11T22:52:28.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:52:28.558 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:52:28.558 INFO:tasks.workunit.client.0.smithi016.stdout:TEST PASSED 2020-03-11T22:52:28.892 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test1' does not exist 2020-03-11T22:52:28.900 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-03-11T22:52:28.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:52:28.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:52:29.448 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test3' does not exist 2020-03-11T22:52:29.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-03-11T22:52:29.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test7 2020-03-11T22:52:29.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-03-11T22:52:30.126 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test7' does not exist 2020-03-11T22:52:30.134 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:52:30.134 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:30.135 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:30.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:30.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:30.135 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:30.445 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:30.445 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-03-11T22:52:30.445 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:52:30.445 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:52:30.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:52:30.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:52:30.446 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:30.446 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:30.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:30.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:30.446 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:30.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:30.452 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:30.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:30.452 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:30.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:30.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:30.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:30.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:30.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:30.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:30.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:30.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:30.458 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:52:30.458 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:52:30.484 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:30.484 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:30.484 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:52:30.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56083 2020-03-11T22:52:30.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:30.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:30.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:52:30.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/osd-backfill-prio 0 2020-03-11T22:52:30.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:52:30.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:52:30.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:52:30.487 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:30.487 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:30.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:30.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:30.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:30.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:30.488 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:30.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:30.489 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:30.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:30.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:30.490 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:30.492 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:30.492 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:30.492 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:30.492 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:30.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:30.493 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:52:30.494 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:52:30.495 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:30.495 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:30.495 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56083 2020-03-11T22:52:30.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56083 2020-03-11T22:52:30.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:30.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:30.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:52:30.497 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:52:30.497 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:52:30.564 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-03-11T22:52:30.564 INFO:teuthology.orchestra.run.smithi016:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-03-11T22:52:30.596 INFO:tasks.workunit.client.0.smithi016.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:52:30.603 INFO:tasks.workunit.client.0.smithi016.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:52:30.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:52:30.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:52:30.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:52:30.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:52:30.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:52:30.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:52:30.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:52:30.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:52:30.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-recovery-log 2020-03-11T22:52:30.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:30.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-03-11T22:52:30.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-03-11T22:52:30.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-03-11T22:52:30.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-03-11T22:52:30.606 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-03-11T22:52:30.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none ' 2020-03-11T22:52:30.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:30.606 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-03-11T22:52:30.606 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:52:30.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-03-11T22:52:30.608 INFO:tasks.workunit.client.0.smithi016.stderr:TEST_backfill_log_2 2020-03-11T22:52:30.608 INFO:tasks.workunit.client.0.smithi016.stderr:TEST_recovery_1 2020-03-11T22:52:30.609 INFO:tasks.workunit.client.0.smithi016.stderr:TEST_recovery_2' 2020-03-11T22:52:30.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:52:30.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:52:30.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:30.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:52:30.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:30.609 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:52:30.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:52:30.610 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:30.610 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:30.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:30.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:30.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:30.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:30.611 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:30.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:30.611 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:30.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:30.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:30.612 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:30.612 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:30.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:30.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:30.613 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:30.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:30.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:52:30.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:52:30.614 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:30.614 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:30.614 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:30.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76802 2020-03-11T22:52:30.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:30.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:30.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:52:30.620 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:52:30.620 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:30.620 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:30.621 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76802 2020-03-11T22:52:30.621 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:52:30.621 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:52:30.621 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:52:30.621 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:52:30.621 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:52:30.621 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-03-11T22:52:30.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:103: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:30.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-03-11T22:52:30.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:30.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:30.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:52:30.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-03-11T22:52:30.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:52:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:52:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:52:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:52:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:52:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:52:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:52:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:52:30.624 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:52:30.939 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:52:30.939 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:30.939 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:30.939 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:30.940 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:30.940 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:30.940 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:30.940 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:52:31.157 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:52:31.159 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:52:31.159 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:52:31.159 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:52:31.159 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:52:31.159 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:52:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:52:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:52:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:52:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76802/ceph-mon.a.asok 2020-03-11T22:52:31.161 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:52:31.161 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76802/ceph-mon.a.asok config get fsid 2020-03-11T22:52:31.255 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:52:31.255 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:52:31.255 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:52:31.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:52:31.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:52:31.256 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:52:31.256 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:52:31.256 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:52:31.256 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:31.256 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:31.257 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:31.257 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76802/ceph-mon.a.asok 2020-03-11T22:52:31.257 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:52:31.257 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76802/ceph-mon.a.asok config get mon_host 2020-03-11T22:52:31.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:52:31.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:31.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:52:31.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:52:31.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:52:31.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:52:31.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:52:32.284 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:52:32.284 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:32.285 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:32.285 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:32.285 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:32.285 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:32.285 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:32.286 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:52:32.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:52:32.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:52:32.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:32.308 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:32.308 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:52:32.309 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:52:32.315 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:32.315 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:52:32.315 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:32.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:32.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:52:32.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:32.316 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:52:32.317 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:32.317 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:32.317 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:32.317 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:32.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:52:32.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:52:32.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:32.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:32.319 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:32.319 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:32.319 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:32.319 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:32.320 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:32.320 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:32.320 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:32.320 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:32.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:32.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:32.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:32.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:32.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:32.322 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:32.322 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:32.322 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:32.322 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:32.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:32.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:32.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:52:32.323 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:32.323 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=5160306a-d8cc-4508-bd32-f8ee7fe900da 2020-03-11T22:52:32.324 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 5160306a-d8cc-4508-bd32-f8ee7fe900da' 2020-03-11T22:52:32.324 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:32.324 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 5160306a-d8cc-4508-bd32-f8ee7fe900da 2020-03-11T22:52:32.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCwa2lezsXoExAALnEEo958PFrV1V3fHq8HOg== 2020-03-11T22:52:32.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCwa2lezsXoExAALnEEo958PFrV1V3fHq8HOg=="}' 2020-03-11T22:52:32.337 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 5160306a-d8cc-4508-bd32-f8ee7fe900da -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:52:32.658 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:52:32.668 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:52:32.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCwa2lezsXoExAALnEEo958PFrV1V3fHq8HOg== --osd-uuid 5160306a-d8cc-4508-bd32-f8ee7fe900da 2020-03-11T22:52:32.694 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:32.690+0000 7f232c154ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:32.696 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:32.692+0000 7f232c154ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:32.697 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:32.694+0000 7f232c154ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:32.698 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:32.694+0000 7f232c154ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:32.698 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:32.694+0000 7f232c154ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:32.698 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:32.694+0000 7f232c154ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:32.790 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:32.786+0000 7f232c154ec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:52:35.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:52:35.893 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:35.894 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:52:35.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:52:35.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:36.370 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:52:36.370 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:52:36.370 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:36.370 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:36.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:36.371 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:36.393 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:36.389+0000 7fe4ecb4dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:36.395 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:36.391+0000 7fe4ecb4dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:36.396 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:36.392+0000 7fe4ecb4dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:36.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:52:36.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:36.735 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:52:36.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:36.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:36.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:36.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:36.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:36.736 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:52:36.737 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:52:36.911 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:36.907+0000 7fe4ecb4dec0 -1 Falling back to public interface 2020-03-11T22:52:37.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:37.698 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:37.695+0000 7fe4ecb4dec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:52:38.096 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:52:38.097 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:52:38.517 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/77234,v1:127.0.0.1:6803/77234] [v2:127.0.0.1:6804/77234,v1:127.0.0.1:6805/77234] exists,up 5160306a-d8cc-4508-bd32-f8ee7fe900da 2020-03-11T22:52:38.517 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 1a35732c-f53f-4fa5-aedd-d1a5c256da7a 2020-03-11T22:52:38.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:38.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:38.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:38.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:38.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:52:38.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:38.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:38.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:52:38.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:38.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:52:38.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:38.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:38.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:38.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:38.521 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:38.521 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:38.521 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:38.521 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:38.521 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:38.521 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:38.521 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:38.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:38.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:38.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:38.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:38.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:38.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:38.522 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:38.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:38.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:38.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:52:38.523 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:38.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1a35732c-f53f-4fa5-aedd-d1a5c256da7a 2020-03-11T22:52:38.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 1a35732c-f53f-4fa5-aedd-d1a5c256da7a' 2020-03-11T22:52:38.523 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:38.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC2a2le7gd6HxAAKr0MpOF7vnesxql0/PDQ/w== 2020-03-11T22:52:38.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC2a2le7gd6HxAAKr0MpOF7vnesxql0/PDQ/w=="}' 2020-03-11T22:52:38.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1a35732c-f53f-4fa5-aedd-d1a5c256da7a -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:52:39.061 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:52:39.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:52:39.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC2a2le7gd6HxAAKr0MpOF7vnesxql0/PDQ/w== --osd-uuid 1a35732c-f53f-4fa5-aedd-d1a5c256da7a 2020-03-11T22:52:39.097 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:39.093+0000 7f7734d83ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:39.114 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:39.110+0000 7f7734d83ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:39.115 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:39.111+0000 7f7734d83ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:39.115 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:39.112+0000 7f7734d83ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:39.115 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:39.112+0000 7f7734d83ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:39.116 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:39.112+0000 7f7734d83ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:39.205 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:39.202+0000 7f7734d83ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:52:42.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:52:42.356 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:42.356 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:52:42.357 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:52:42.357 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:42.845 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:52:42.845 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:52:42.845 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:42.845 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:42.845 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:42.846 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:42.868 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:42.864+0000 7f7a60fc5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:42.869 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:42.866+0000 7f7a60fc5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:42.871 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:42.867+0000 7f7a60fc5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:43.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:52:43.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:43.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:52:43.212 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:43.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:43.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:43.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:43.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:43.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:43.213 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:52:43.386 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:43.382+0000 7f7a60fc5ec0 -1 Falling back to public interface 2020-03-11T22:52:43.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:44.164 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:44.161+0000 7f7a60fc5ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:52:44.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:44.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:44.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:44.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:44.580 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:44.580 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:52:44.958 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/77589,v1:127.0.0.1:6811/77589] [v2:127.0.0.1:6812/77589,v1:127.0.0.1:6813/77589] exists,up 1a35732c-f53f-4fa5-aedd-d1a5c256da7a 2020-03-11T22:52:44.958 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:44.958 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:44.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:44.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:44.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:52:44.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:44.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:44.959 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:52:44.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:44.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:52:44.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:44.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:44.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:44.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:44.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:52:44.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:52:44.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:44.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:44.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:44.962 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:44.962 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:44.962 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:44.962 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:44.962 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:44.963 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:44.963 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:44.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:44.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:44.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:44.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:44.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:44.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:44.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:44.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:44.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:44.964 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:44.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:44.965 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:52:44.965 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:44.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6079daf7-1217-4091-800f-f8696309b5ee 2020-03-11T22:52:44.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 6079daf7-1217-4091-800f-f8696309b5ee' 2020-03-11T22:52:44.966 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:44.966 INFO:tasks.workunit.client.0.smithi016.stdout:add osd2 6079daf7-1217-4091-800f-f8696309b5ee 2020-03-11T22:52:44.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC8a2le99/fORAAko4rpF4xGlHoQbcCBAzdGw== 2020-03-11T22:52:44.973 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC8a2le99/fORAAko4rpF4xGlHoQbcCBAzdGw=="}' 2020-03-11T22:52:44.974 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6079daf7-1217-4091-800f-f8696309b5ee -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:52:45.525 INFO:tasks.workunit.client.0.smithi016.stdout:2 2020-03-11T22:52:45.535 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:52:45.536 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC8a2le99/fORAAko4rpF4xGlHoQbcCBAzdGw== --osd-uuid 6079daf7-1217-4091-800f-f8696309b5ee 2020-03-11T22:52:45.560 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:45.557+0000 7f11a738eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:45.579 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:45.576+0000 7f11a738eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:45.581 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:45.577+0000 7f11a738eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:45.582 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:45.578+0000 7f11a738eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:45.582 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:45.578+0000 7f11a738eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:45.582 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:45.578+0000 7f11a738eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:45.704 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:45.701+0000 7f11a738eec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:52:49.078 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:52:49.089 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:49.090 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd2 key to auth repository 2020-03-11T22:52:49.090 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:52:49.090 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:49.561 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:52:49.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:52:49.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:49.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:49.562 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:49.566 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:49.584 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:49.581+0000 7f2647288ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:49.586 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:49.583+0000 7f2647288ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:49.588 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:49.584+0000 7f2647288ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:49.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:52:49.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:49.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:52:49.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:49.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:49.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:49.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:49.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:49.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:49.955 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:52:50.113 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:50.109+0000 7f2647288ec0 -1 Falling back to public interface 2020-03-11T22:52:50.373 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:50.913 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:50.910+0000 7f2647288ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:52:51.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:51.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:51.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:51.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:51.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:51.376 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:52:51.766 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/77944,v1:127.0.0.1:6819/77944] [v2:127.0.0.1:6820/77944,v1:127.0.0.1:6821/77944] exists,up 6079daf7-1217-4091-800f-f8696309b5ee 2020-03-11T22:52:51.766 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:51.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:51.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:51.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:51.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:52:51.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:51.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:51.767 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:52:51.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:51.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:52:51.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:51.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:51.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:51.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:51.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:52:51.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:52:51.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:51.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:51.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:51.769 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:51.769 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:51.769 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:51.770 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:51.770 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:51.770 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:51.770 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:51.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:51.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:51.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:51.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:51.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:51.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:51.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:51.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:51.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:51.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:51.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:51.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:52:51.773 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:51.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e18d325d-16e4-459f-87d8-d95984ddd63a 2020-03-11T22:52:51.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 e18d325d-16e4-459f-87d8-d95984ddd63a' 2020-03-11T22:52:51.773 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:51.773 INFO:tasks.workunit.client.0.smithi016.stdout:add osd3 e18d325d-16e4-459f-87d8-d95984ddd63a 2020-03-11T22:52:51.782 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDDa2leARF6LhAAHfmGtAMSSekgd8TwZsm14A== 2020-03-11T22:52:51.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDDa2leARF6LhAAHfmGtAMSSekgd8TwZsm14A=="}' 2020-03-11T22:52:51.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e18d325d-16e4-459f-87d8-d95984ddd63a -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:52:52.268 INFO:tasks.workunit.client.0.smithi016.stdout:3 2020-03-11T22:52:52.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:52:52.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDDa2leARF6LhAAHfmGtAMSSekgd8TwZsm14A== --osd-uuid e18d325d-16e4-459f-87d8-d95984ddd63a 2020-03-11T22:52:52.304 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:52.300+0000 7f369dfc7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:52.305 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:52.302+0000 7f369dfc7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:52.307 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:52.303+0000 7f369dfc7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:52.307 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:52.304+0000 7f369dfc7ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:52.307 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:52.304+0000 7f369dfc7ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:52.308 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:52.304+0000 7f369dfc7ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:52.371 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:52.368+0000 7f369dfc7ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:52:55.551 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:52:55.551 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:55.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:52:55.552 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:55.553 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd3 key to auth repository 2020-03-11T22:52:56.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:52:56.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:56.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:56.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:56.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:56.045 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.3 2020-03-11T22:52:56.067 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:56.063+0000 7f23fd325ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:56.068 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:56.065+0000 7f23fd325ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:56.070 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:56.067+0000 7f23fd325ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:56.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:52:56.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:56.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:52:56.496 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:56.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:56.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:56.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:56.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:56.497 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:56.497 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:52:56.596 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:56.592+0000 7f23fd325ec0 -1 Falling back to public interface 2020-03-11T22:52:56.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:57.389 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:57.385+0000 7f23fd325ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:52:57.945 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:57.945 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:57.945 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:57.945 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:57.945 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:57.946 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:52:58.451 INFO:tasks.workunit.client.0.smithi016.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/78325,v1:127.0.0.1:6827/78325] [v2:127.0.0.1:6828/78325,v1:127.0.0.1:6829/78325] exists,up e18d325d-16e4-459f-87d8-d95984ddd63a 2020-03-11T22:52:58.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:58.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:58.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:58.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:58.451 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:52:58.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:58.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:58.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:52:58.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:58.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:52:58.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:58.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:58.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:58.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:58.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:52:58.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:52:58.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:58.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:58.454 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:58.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:58.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:58.454 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:58.454 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:58.454 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:58.454 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:52:58.455 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:58.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:52:58.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:58.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:58.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:58.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:58.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:58.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:58.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:58.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:58.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:58.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:58.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:52:58.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:58.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=5dc0807f-2061-49cd-821d-578ae7f17e97 2020-03-11T22:52:58.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 5dc0807f-2061-49cd-821d-578ae7f17e97' 2020-03-11T22:52:58.458 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:58.458 INFO:tasks.workunit.client.0.smithi016.stdout:add osd4 5dc0807f-2061-49cd-821d-578ae7f17e97 2020-03-11T22:52:58.466 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDKa2lenSmkGxAA3ERP91G7xwfD/oJspYamKw== 2020-03-11T22:52:58.466 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDKa2lenSmkGxAA3ERP91G7xwfD/oJspYamKw=="}' 2020-03-11T22:52:58.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 5dc0807f-2061-49cd-821d-578ae7f17e97 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:52:58.943 INFO:tasks.workunit.client.0.smithi016.stdout:4 2020-03-11T22:52:58.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:52:58.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDKa2lenSmkGxAA3ERP91G7xwfD/oJspYamKw== --osd-uuid 5dc0807f-2061-49cd-821d-578ae7f17e97 2020-03-11T22:52:58.979 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:58.975+0000 7f7fe7e41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:58.980 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:58.977+0000 7f7fe7e41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:58.982 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:58.978+0000 7f7fe7e41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:58.982 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:58.979+0000 7f7fe7e41ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:58.983 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:58.979+0000 7f7fe7e41ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:58.983 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:58.979+0000 7f7fe7e41ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:59.014 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:52:59.011+0000 7f7fe7e41ec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:53:02.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:53:02.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:02.300 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd4 key to auth repository 2020-03-11T22:53:02.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:53:02.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:02.792 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.4 2020-03-11T22:53:02.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:53:02.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:02.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:02.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:02.793 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:02.815 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:02.813+0000 7f82b9a41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:02.817 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:02.815+0000 7f82b9a41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:02.819 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:02.816+0000 7f82b9a41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:03.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:53:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:53:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:03.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:03.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:53:03.163 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:53:03.334 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:03.331+0000 7f82b9a41ec0 -1 Falling back to public interface 2020-03-11T22:53:03.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:04.116 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:04.114+0000 7f82b9a41ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:53:04.532 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:04.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:04.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:04.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:04.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:53:04.533 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:53:04.917 INFO:tasks.workunit.client.0.smithi016.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/78680,v1:127.0.0.1:6835/78680] [v2:127.0.0.1:6836/78680,v1:127.0.0.1:6837/78680] exists,up 5dc0807f-2061-49cd-821d-578ae7f17e97 2020-03-11T22:53:04.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:04.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:04.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:04.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:04.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:53:04.918 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:04.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:04.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:53:04.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:04.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:53:04.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:04.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:04.919 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:04.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:04.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:53:04.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:53:04.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:04.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:04.920 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:04.920 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:04.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:04.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:04.921 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:04.921 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:04.921 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:53:04.921 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:53:04.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:53:04.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:04.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:04.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:04.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:04.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:04.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:04.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:04.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:04.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:04.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:04.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:53:04.924 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:04.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9529c76d-3529-4ebd-8f5f-ad1d79049fd1 2020-03-11T22:53:04.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 9529c76d-3529-4ebd-8f5f-ad1d79049fd1' 2020-03-11T22:53:04.924 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:04.924 INFO:tasks.workunit.client.0.smithi016.stdout:add osd5 9529c76d-3529-4ebd-8f5f-ad1d79049fd1 2020-03-11T22:53:04.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDQa2leX/d9NxAA50tdXcakgLkckaZRbS/rRQ== 2020-03-11T22:53:04.934 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDQa2leX/d9NxAA50tdXcakgLkckaZRbS/rRQ=="}' 2020-03-11T22:53:04.934 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9529c76d-3529-4ebd-8f5f-ad1d79049fd1 -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:53:05.520 INFO:tasks.workunit.client.0.smithi016.stdout:5 2020-03-11T22:53:05.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:53:05.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQa2leX/d9NxAA50tdXcakgLkckaZRbS/rRQ== --osd-uuid 9529c76d-3529-4ebd-8f5f-ad1d79049fd1 2020-03-11T22:53:05.556 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:05.553+0000 7f5e42c92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:05.558 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:05.555+0000 7f5e42c92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:05.559 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:05.556+0000 7f5e42c92ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:05.559 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:05.557+0000 7f5e42c92ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:05.560 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:05.557+0000 7f5e42c92ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:05.560 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:05.557+0000 7f5e42c92ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:05.620 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:05.617+0000 7f5e42c92ec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:53:09.094 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:53:09.094 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:09.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:53:09.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:09.095 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd5 key to auth repository 2020-03-11T22:53:09.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:53:09.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:09.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:09.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:09.583 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:09.583 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.5 2020-03-11T22:53:09.604 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:09.601+0000 7f5d3b448ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:09.606 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:09.603+0000 7f5d3b448ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:09.608 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:09.605+0000 7f5d3b448ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:10.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:53:10.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:10.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:53:10.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:10.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:10.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:10.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:10.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:10.059 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:10.059 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:53:10.126 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:10.124+0000 7f5d3b448ec0 -1 Falling back to public interface 2020-03-11T22:53:10.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:10.890 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:53:10.887+0000 7f5d3b448ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:53:11.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:11.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:11.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:11.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:11.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:11.458 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:53:11.832 INFO:tasks.workunit.client.0.smithi016.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/79035,v1:127.0.0.1:6843/79035] [v2:127.0.0.1:6844/79035,v1:127.0.0.1:6845/79035] exists,up 9529c76d-3529-4ebd-8f5f-ad1d79049fd1 2020-03-11T22:53:11.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:11.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:11.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:11.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:53:11.833 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:53:12.367 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' already exists 2020-03-11T22:53:12.375 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:53:13.377 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:53:13.378 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.378 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:53:13.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.582 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:53:13.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.628 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:53:13.693 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.694 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:53:13.841 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.843 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:53:13.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:53:13.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:13.998 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:53:14.061 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.062 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:53:14.167 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.167 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:53:14.303 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.303 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:53:14.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.345 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:53:14.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:53:14.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.495 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:53:14.540 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.540 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:53:14.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:53:14.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:53:14.715 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.715 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:53:14.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:53:14.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.861 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:53:14.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:53:14.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:53:14.970 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:14.970 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:53:15.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.111 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:53:15.148 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.149 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:53:15.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:53:15.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.307 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:53:15.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:53:15.387 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.387 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:53:15.420 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.420 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:53:15.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.520 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:53:15.590 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:53:15.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.629 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:53:15.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:53:15.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:53:15.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:53:15.866 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:15.866 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:53:16.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:53:16.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.044 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:53:16.079 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.079 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:53:16.205 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.206 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:53:16.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.245 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:53:16.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:53:16.313 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.313 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:53:16.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:53:16.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:53:16.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:53:16.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.757 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:53:16.865 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.865 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:53:16.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:53:16.938 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:16.938 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:53:17.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.074 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:53:17.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:53:17.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.146 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:53:17.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:53:17.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:53:17.380 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.380 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:53:17.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.426 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:53:17.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:53:17.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:53:17.613 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:53:17.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:53:17.733 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.734 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:53:17.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:53:17.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:17.834 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:53:18.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.018 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:53:18.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.110 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:53:18.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.147 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:53:18.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.180 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:53:18.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:53:18.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:53:18.380 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.381 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:53:18.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:53:18.509 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.510 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:53:18.547 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.548 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:53:18.581 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.581 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:53:18.744 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.745 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:53:18.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:53:18.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.822 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:53:18.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:53:18.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:18.993 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:53:19.032 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.032 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:53:19.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.064 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:53:19.182 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:53:19.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.226 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:53:19.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:53:19.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.346 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:53:19.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:53:19.501 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:53:19.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.539 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:53:19.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:53:19.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:53:19.699 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.699 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:53:19.829 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.829 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:53:19.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:19.957 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:53:20.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.023 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:53:20.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:53:20.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:53:20.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:53:20.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.215 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:53:20.382 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.383 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:53:20.466 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.466 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:53:20.510 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.510 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:53:20.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:53:20.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.647 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:53:20.681 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.682 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:53:20.758 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.759 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:53:20.875 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.876 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:53:20.913 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.914 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:53:20.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:53:21.068 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.068 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:53:21.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.127 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:53:21.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.166 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:53:21.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:53:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:53:21.394 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.395 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:53:21.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:53:21.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:53:21.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.557 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:53:21.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:53:21.656 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:53:21.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.761 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:53:21.799 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.799 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:53:21.841 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.841 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:53:21.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:53:22.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.016 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:53:22.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:53:22.083 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.083 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:53:22.230 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.230 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:53:22.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:53:22.299 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.300 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:53:22.380 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.380 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:53:22.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:53:22.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:53:22.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:53:22.614 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:53:22.653 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.653 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:53:22.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.691 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:53:22.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:53:22.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.890 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:53:22.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.937 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:53:22.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:53:23.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.116 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:53:23.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.154 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:53:23.191 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:53:23.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:53:23.395 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.396 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:53:23.433 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.434 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:53:23.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.467 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:53:23.601 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:53:23.601 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:53:23.945 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:53:23.953 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:53:24.479 INFO:tasks.workunit.client.0.smithi016.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:53:24.486 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:53:24.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:53:25.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:53:25.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:53:25.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:53:25.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:53:25.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:53:25.490 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:53:25.490 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:53:25.490 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:53:25.490 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:53:25.490 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:53:25.490 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:53:25.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:53:25.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:53:25.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:53:25.595 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:25.596 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:25.982 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:53:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:53:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:53:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:4 2020-03-11T22:53:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:5' 2020-03-11T22:53:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:25.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:25.984 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:26.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:53:26.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:53:26.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:26.113 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:26.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:53:26.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2020-03-11T22:53:26.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:26.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:26.385 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-03-11T22:53:26.385 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449' 2020-03-11T22:53:26.385 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:26.385 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:26.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345928 2020-03-11T22:53:26.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928' 2020-03-11T22:53:26.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:26.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:26.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182407 2020-03-11T22:53:26.644 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928 4-107374182407' 2020-03-11T22:53:26.645 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:26.645 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:53:26.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018885 2020-03-11T22:53:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928 4-107374182407 5-128849018885' 2020-03-11T22:53:26.774 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:26.774 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:53:26.774 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:26.775 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:26.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:53:26.776 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:26.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:53:26.777 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:53:26.778 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:53:26.778 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:27.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836492 2020-03-11T22:53:27.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:28.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:28.144 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:28.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:53:28.527 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:28.528 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:53:28.528 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:28.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:28.530 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:53:28.531 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:28.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:53:28.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:53:28.532 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:53:28.532 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:28.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672971 2020-03-11T22:53:28.894 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:28.895 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509449 2020-03-11T22:53:28.895 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:28.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:28.897 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509449 2020-03-11T22:53:28.897 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:28.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509449 2020-03-11T22:53:28.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-03-11T22:53:28.898 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509449 2020-03-11T22:53:28.898 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:29.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509449 2020-03-11T22:53:29.265 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:29.266 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345928 2020-03-11T22:53:29.266 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:29.266 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:29.267 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345928 2020-03-11T22:53:29.268 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:29.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345928 2020-03-11T22:53:29.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2020-03-11T22:53:29.269 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345928 2020-03-11T22:53:29.269 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:29.747 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345928 -lt 85899345928 2020-03-11T22:53:29.747 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:29.748 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182407 2020-03-11T22:53:29.748 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:29.749 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:29.750 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182407 2020-03-11T22:53:29.750 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:29.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182407 2020-03-11T22:53:29.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182407' 2020-03-11T22:53:29.752 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182407 2020-03-11T22:53:29.752 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:30.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182407 -lt 107374182407 2020-03-11T22:53:30.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:30.140 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-128849018885 2020-03-11T22:53:30.141 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:30.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:53:30.142 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-128849018885 2020-03-11T22:53:30.142 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:30.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=128849018885 2020-03-11T22:53:30.143 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 128849018885' 2020-03-11T22:53:30.144 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.5 seq 128849018885 2020-03-11T22:53:30.144 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:53:30.514 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 128849018886 -lt 128849018885 2020-03-11T22:53:30.515 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:53:30.515 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:30.515 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:31.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:53:31.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:31.004 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:31.005 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:31.005 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:31.005 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:31.005 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:31.005 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:31.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:31.356 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:31.356 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:31.357 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:31.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:31.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:53:31.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:53:31.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:53:31.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:31.831 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:31.931 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:31.931 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:31.933 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:31.933 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:31.933 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:31.933 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:31.933 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:31.933 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:32.515 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:32.515 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:32.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:32.516 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:32.991 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:32.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:32.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:32.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:32.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:32.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:32.992 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:32.993 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:32.993 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:33.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14888 2020-03-11T22:53:33.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14888 '!=' null 2020-03-11T22:53:33.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:33.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:33.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:33.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:33.591 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:33.592 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:33.593 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:33.593 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:33.593 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:33.593 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:33.593 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:33.943 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:33.943 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:33.944 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:33.944 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:34.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:34.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:34.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:34.421 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:34.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:34.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:34.422 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:34.422 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:34.423 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:34.898 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=12620 2020-03-11T22:53:34.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 12620 '!=' null 2020-03-11T22:53:34.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:34.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:34.899 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:34.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:34.999 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:35.000 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:35.000 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:35.000 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:35.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:35.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:35.001 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:35.352 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:35.352 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:35.353 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:35.353 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:35.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:35.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:35.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:35.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:35.824 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:35.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:35.825 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:35.825 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:35.825 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:36.354 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27632 2020-03-11T22:53:36.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27632 '!=' null 2020-03-11T22:53:36.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:36.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:36.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:36.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:36.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:36.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:36.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:36.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:36.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:36.456 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:36.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:36.810 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:36.811 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:36.811 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:36.811 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:37.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:37.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:37.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:37.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:37.282 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:37.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:37.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:37.283 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:37.283 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:37.791 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27632 2020-03-11T22:53:37.792 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27632 '!=' null 2020-03-11T22:53:37.792 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:37.792 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:37.792 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:37.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:37.892 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:37.893 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:37.893 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:37.893 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:37.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:37.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:37.894 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:38.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:38.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:38.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:38.271 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:38.741 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:38.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:38.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:38.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:38.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:38.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:38.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:38.743 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:38.743 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:39.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24148 2020-03-11T22:53:39.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24148 '!=' null 2020-03-11T22:53:39.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:39.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:39.298 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:39.398 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:39.398 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:39.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:39.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:39.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:39.399 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:39.400 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:39.400 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:39.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:39.753 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:39.753 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:39.753 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:40.250 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:40.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:40.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:40.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:40.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:40.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:40.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:40.252 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:40.252 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:40.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=38523 2020-03-11T22:53:40.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 38523 '!=' null 2020-03-11T22:53:40.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:40.728 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:40.729 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:40.829 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:40.829 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:40.830 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:40.830 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:40.830 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:40.831 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:40.831 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:40.831 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:41.183 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:41.184 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:41.184 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:41.184 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:41.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:41.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:41.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:41.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:41.657 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:41.658 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:41.658 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:41.658 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:41.658 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:42.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29324 2020-03-11T22:53:42.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29324 '!=' null 2020-03-11T22:53:42.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:42.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:42.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:42.265 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:42.265 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:42.266 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:42.267 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:42.267 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:42.267 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:42.267 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:42.267 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:42.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:42.624 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:42.624 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:42.625 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:43.098 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:43.098 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:43.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:43.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:43.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:43.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:43.099 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:43.100 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:43.100 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:43.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29324 2020-03-11T22:53:43.601 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29324 '!=' null 2020-03-11T22:53:43.602 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:43.602 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:43.602 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:43.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:43.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:43.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:43.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:43.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:43.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:43.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:43.703 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:44.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:44.057 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:44.058 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:44.058 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:44.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:44.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:44.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:44.529 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:44.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:44.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:44.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:44.530 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:44.531 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:45.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29324 2020-03-11T22:53:45.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29324 '!=' null 2020-03-11T22:53:45.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:45.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:45.040 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:45.140 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:45.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:45.142 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:45.142 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:45.143 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:45.143 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:45.143 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:45.143 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:45.491 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:45.492 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:45.492 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:45.493 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:45.966 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:45.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:45.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:45.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:45.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:45.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:45.967 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:45.968 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:45.968 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:46.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=44849 2020-03-11T22:53:46.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 44849 '!=' null 2020-03-11T22:53:46.443 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:46.443 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:46.443 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:46.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:46.543 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:46.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:46.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:46.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:46.545 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:46.545 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:46.545 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:46.900 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:46.901 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:46.901 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:46.901 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:47.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:47.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:47.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:47.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:47.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:47.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:47.405 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:47.405 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:47.405 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:47.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=44849 2020-03-11T22:53:47.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 44849 '!=' null 2020-03-11T22:53:47.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:47.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:47.877 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:47.977 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:47.978 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:47.979 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:47.979 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:47.980 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:47.980 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:47.980 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:47.980 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:48.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:48.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:48.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:48.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:48.805 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:48.806 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:49.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29899 2020-03-11T22:53:49.285 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29899 '!=' null 2020-03-11T22:53:49.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:49.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:49.286 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:49.385 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:49.386 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:49.387 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:49.388 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:49.388 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:49.388 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:49.388 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:49.389 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:49.742 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:49.743 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:49.743 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:49.743 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:50.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:50.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:50.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:50.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:50.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:50.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:50.240 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:50.241 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:50.241 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:50.713 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45423 2020-03-11T22:53:50.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45423 '!=' null 2020-03-11T22:53:50.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:50.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:50.714 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:50.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:50.814 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:50.816 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:50.816 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:50.816 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:50.816 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:50.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:50.817 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:51.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:51.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:51.640 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:51.641 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:51.641 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:51.641 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:51.641 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:51.641 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:51.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:51.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:52.123 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:53:52.123 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:53:52.123 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:52.123 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:52.123 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:52.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:52.224 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:52.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:52.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:52.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:52.227 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:52.228 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:52.228 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:52.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:52.577 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:52.577 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:52.577 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:53.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:53.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:53.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:53.053 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:53.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:53.054 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:53.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:53:53.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:53:53.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:53.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:53.525 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:53.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:53.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:53.627 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:53.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:53.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:53.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:53.628 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:53.629 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:53.981 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:53.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:53.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:53.982 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:54.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:54.472 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:54.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:54.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:54.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:54.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:54.473 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:54.474 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:54.474 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:54.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:53:54.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:53:54.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:54.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:54.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:55.049 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:55.050 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:55.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:55.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:55.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:55.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:55.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:55.051 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:55.403 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:55.404 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:55.404 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:55.404 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:55.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:55.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:55.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:55.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:55.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:55.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:55.884 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:55.885 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:55.885 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:56.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=48297 2020-03-11T22:53:56.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 48297 '!=' null 2020-03-11T22:53:56.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:56.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:56.355 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:56.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:56.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:56.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:56.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:56.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:56.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:56.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:56.457 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:56.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:53:56.807 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:56.807 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:56.807 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:57.283 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:53:57.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:53:57.284 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:53:57.284 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:53:57.285 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:53:57.662 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:53:57.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:53:57.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:53:57.671 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:57.672 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:57.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:57.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:57.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:57.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:57.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:53:57.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:53:57.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:57.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:57.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:53:57.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:53:57.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:53:57.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:53:57.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:53:58.879 INFO:tasks.workunit.client.0.smithi016.stdout:{ 2020-03-11T22:53:58.880 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_log_t": { 2020-03-11T22:53:58.880 INFO:tasks.workunit.client.0.smithi016.stdout: "head": "34'150", 2020-03-11T22:53:58.880 INFO:tasks.workunit.client.0.smithi016.stdout: "tail": "34'148", 2020-03-11T22:53:58.880 INFO:tasks.workunit.client.0.smithi016.stdout: "log": [ 2020-03-11T22:53:58.880 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.880 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "modify", 2020-03-11T22:53:58.880 INFO:tasks.workunit.client.0.smithi016.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-03-11T22:53:58.881 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'149", 2020-03-11T22:53:58.881 INFO:tasks.workunit.client.0.smithi016.stdout: "prior_version": "0'0", 2020-03-11T22:53:58.881 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4631.0:1", 2020-03-11T22:53:58.881 INFO:tasks.workunit.client.0.smithi016.stdout: "extra_reqids": [], 2020-03-11T22:53:58.881 INFO:tasks.workunit.client.0.smithi016.stdout: "mtime": "2020-03-11T22:53:23.453769+0000", 2020-03-11T22:53:58.881 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": 0, 2020-03-11T22:53:58.882 INFO:tasks.workunit.client.0.smithi016.stdout: "mod_desc": { 2020-03-11T22:53:58.882 INFO:tasks.workunit.client.0.smithi016.stdout: "object_mod_desc": { 2020-03-11T22:53:58.882 INFO:tasks.workunit.client.0.smithi016.stdout: "can_local_rollback": false, 2020-03-11T22:53:58.882 INFO:tasks.workunit.client.0.smithi016.stdout: "rollback_info_completed": false, 2020-03-11T22:53:58.882 INFO:tasks.workunit.client.0.smithi016.stdout: "ops": [] 2020-03-11T22:53:58.882 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.882 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.883 INFO:tasks.workunit.client.0.smithi016.stdout: "clean_regions": { 2020-03-11T22:53:58.883 INFO:tasks.workunit.client.0.smithi016.stdout: "object_clean_regions": { 2020-03-11T22:53:58.883 INFO:tasks.workunit.client.0.smithi016.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:53:58.883 INFO:tasks.workunit.client.0.smithi016.stdout: "clean_omap": true, 2020-03-11T22:53:58.883 INFO:tasks.workunit.client.0.smithi016.stdout: "new_object": false 2020-03-11T22:53:58.883 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.883 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.884 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.884 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.884 INFO:tasks.workunit.client.0.smithi016.stdout: "op": "modify", 2020-03-11T22:53:58.884 INFO:tasks.workunit.client.0.smithi016.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-03-11T22:53:58.884 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'150", 2020-03-11T22:53:58.884 INFO:tasks.workunit.client.0.smithi016.stdout: "prior_version": "0'0", 2020-03-11T22:53:58.884 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4634.0:1", 2020-03-11T22:53:58.884 INFO:tasks.workunit.client.0.smithi016.stdout: "extra_reqids": [], 2020-03-11T22:53:58.885 INFO:tasks.workunit.client.0.smithi016.stdout: "mtime": "2020-03-11T22:53:23.487266+0000", 2020-03-11T22:53:58.885 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": 0, 2020-03-11T22:53:58.885 INFO:tasks.workunit.client.0.smithi016.stdout: "mod_desc": { 2020-03-11T22:53:58.885 INFO:tasks.workunit.client.0.smithi016.stdout: "object_mod_desc": { 2020-03-11T22:53:58.885 INFO:tasks.workunit.client.0.smithi016.stdout: "can_local_rollback": false, 2020-03-11T22:53:58.885 INFO:tasks.workunit.client.0.smithi016.stdout: "rollback_info_completed": false, 2020-03-11T22:53:58.886 INFO:tasks.workunit.client.0.smithi016.stdout: "ops": [] 2020-03-11T22:53:58.886 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "clean_regions": { 2020-03-11T22:53:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "object_clean_regions": { 2020-03-11T22:53:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:53:58.887 INFO:tasks.workunit.client.0.smithi016.stdout: "clean_omap": true, 2020-03-11T22:53:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: "new_object": false 2020-03-11T22:53:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.888 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.889 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.889 INFO:tasks.workunit.client.0.smithi016.stdout: ], 2020-03-11T22:53:58.889 INFO:tasks.workunit.client.0.smithi016.stdout: "dups": [ 2020-03-11T22:53:58.889 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.889 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4607.0:1", 2020-03-11T22:53:58.889 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'141", 2020-03-11T22:53:58.890 INFO:tasks.workunit.client.0.smithi016.stdout: "user_version": "141", 2020-03-11T22:53:58.890 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": "0" 2020-03-11T22:53:58.890 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.890 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.890 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4610.0:1", 2020-03-11T22:53:58.890 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'142", 2020-03-11T22:53:58.891 INFO:tasks.workunit.client.0.smithi016.stdout: "user_version": "142", 2020-03-11T22:53:58.891 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": "0" 2020-03-11T22:53:58.891 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.891 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.891 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4613.0:1", 2020-03-11T22:53:58.891 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'143", 2020-03-11T22:53:58.892 INFO:tasks.workunit.client.0.smithi016.stdout: "user_version": "143", 2020-03-11T22:53:58.892 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": "0" 2020-03-11T22:53:58.892 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.892 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.893 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4616.0:1", 2020-03-11T22:53:58.893 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'144", 2020-03-11T22:53:58.893 INFO:tasks.workunit.client.0.smithi016.stdout: "user_version": "144", 2020-03-11T22:53:58.893 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": "0" 2020-03-11T22:53:58.893 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.894 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.894 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4619.0:1", 2020-03-11T22:53:58.894 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'145", 2020-03-11T22:53:58.894 INFO:tasks.workunit.client.0.smithi016.stdout: "user_version": "145", 2020-03-11T22:53:58.894 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": "0" 2020-03-11T22:53:58.895 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.895 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.895 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4622.0:1", 2020-03-11T22:53:58.895 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'146", 2020-03-11T22:53:58.895 INFO:tasks.workunit.client.0.smithi016.stdout: "user_version": "146", 2020-03-11T22:53:58.896 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": "0" 2020-03-11T22:53:58.896 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.896 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.896 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4625.0:1", 2020-03-11T22:53:58.896 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'147", 2020-03-11T22:53:58.897 INFO:tasks.workunit.client.0.smithi016.stdout: "user_version": "147", 2020-03-11T22:53:58.897 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": "0" 2020-03-11T22:53:58.897 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.897 INFO:tasks.workunit.client.0.smithi016.stdout: { 2020-03-11T22:53:58.897 INFO:tasks.workunit.client.0.smithi016.stdout: "reqid": "client.4628.0:1", 2020-03-11T22:53:58.898 INFO:tasks.workunit.client.0.smithi016.stdout: "version": "34'148", 2020-03-11T22:53:58.898 INFO:tasks.workunit.client.0.smithi016.stdout: "user_version": "148", 2020-03-11T22:53:58.898 INFO:tasks.workunit.client.0.smithi016.stdout: "return_code": "0" 2020-03-11T22:53:58.898 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.898 INFO:tasks.workunit.client.0.smithi016.stdout: ] 2020-03-11T22:53:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: }, 2020-03-11T22:53:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: "pg_missing_t": { 2020-03-11T22:53:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: "missing": [], 2020-03-11T22:53:58.899 INFO:tasks.workunit.client.0.smithi016.stdout: "may_include_deletes": true 2020-03-11T22:53:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: } 2020-03-11T22:53:58.900 INFO:tasks.workunit.client.0.smithi016.stdout:} 2020-03-11T22:53:58.902 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:58.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=2 2020-03-11T22:53:58.903 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 2 '!=' 2 ']' 2020-03-11T22:53:58.903 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:58.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=8 2020-03-11T22:53:58.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 8 '!=' 8 ']' 2020-03-11T22:53:58.904 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:53:58.904 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to max 2 2020-03-11T22:53:58.905 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to copy log version 34'150 2020-03-11T22:53:58.905 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to copy log version 34'149 2020-03-11T22:53:58.905 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:58.906 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:53:58.906 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:53:58.906 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:53:58.906 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:53:58.907 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:53:58.907 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:53:58.907 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:53:58.907 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:53:58.908 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to max 2 2020-03-11T22:53:58.908 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to copy log version 34'150 2020-03-11T22:53:58.908 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to copy log version 34'149 2020-03-11T22:53:58.909 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:58.909 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:53:58.909 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:53:58.910 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:53:58.910 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:53:58.910 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:53:58.910 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:53:58.911 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:53:58.911 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:53:58.911 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to max 2 2020-03-11T22:53:58.911 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to copy log version 34'150 2020-03-11T22:53:58.912 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to copy log version 34'149 2020-03-11T22:53:58.912 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:58.912 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:53:58.912 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:53:58.912 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:53:58.915 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:53:58.915 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:53:58.915 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:53:58.916 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:53:58.916 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:25.287+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:53:58.916 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_after v 34'150 2020-03-11T22:53:58.917 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:58.918 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:53:58.918 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:53:58.918 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:53:58.919 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:53:58.919 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:53:58.919 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:53:58.919 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:53:58.920 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:53:58.920 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'149 2020-03-11T22:53:58.920 INFO:tasks.workunit.client.0.smithi016.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:54.873+0000 7fe4cad08700 20 copy_up_to/copy_after copy dup from log version 34'150 2020-03-11T22:53:58.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:58.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:53:58.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:53:58.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:58.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:58.921 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:53:58.922 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:58.922 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:58.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:58.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:58.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:58.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:58.924 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:58.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:58.924 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:58.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:58.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:58.924 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:58.924 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:58.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:58.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:58.925 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:58.925 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:58.926 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:58.926 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:53:58.948 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:58.948 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:58.948 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:53:58.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76802 2020-03-11T22:53:58.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:58.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:58.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:53:58.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:53:58.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:58.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:53:58.949 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:58.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:58.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:53:58.950 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:58.950 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:58.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:58.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:58.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:58.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:58.951 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:58.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:58.951 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:58.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:58.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:58.954 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:58.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:58.954 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:58.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:58.955 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:58.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:58.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:58.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:53:58.955 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:58.955 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:58.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:53:58.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76802 2020-03-11T22:53:58.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:58.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:58.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:53:58.959 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:53:58.959 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:58.959 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:53:58.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76802 2020-03-11T22:53:58.960 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:53:58.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:53:58.960 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:53:58.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:53:58.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2020-03-11T22:53:58.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:111: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:58.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2020-03-11T22:53:58.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:58.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:58.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:53:58.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2020-03-11T22:53:58.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:53:58.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:53:58.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:53:58.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:53:58.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:58.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:53:58.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:53:58.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:53:58.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:53:58.963 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.268 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:53:59.268 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:59.268 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:59.268 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:59.268 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.269 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:53:59.269 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:53:59.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:53:59.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:53:59.458 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:53:59.459 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:53:59.460 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:59.460 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.460 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:53:59.460 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76802/ceph-mon.a.asok 2020-03-11T22:53:59.461 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:53:59.461 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76802/ceph-mon.a.asok config get fsid 2020-03-11T22:53:59.559 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:53:59.559 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:53:59.559 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:53:59.559 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:53:59.559 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:53:59.560 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:53:59.560 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:53:59.560 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:53:59.560 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:59.560 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.560 INFO:tasks.workunit.client.0.smithi016.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:53:59.560 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76802/ceph-mon.a.asok 2020-03-11T22:53:59.561 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:53:59.561 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76802/ceph-mon.a.asok config get mon_host 2020-03-11T22:53:59.651 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:53:59.651 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.651 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:53:59.651 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:53:59.651 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:53:59.651 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:53:59.652 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:54:00.542 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:54:00.542 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:00.543 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:00.543 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:00.543 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:00.543 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:54:00.543 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:00.544 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:54:00.544 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:54:00.567 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:54:00.567 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:00.567 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:00.568 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:54:00.569 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:54:00.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:00.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:54:00.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:00.574 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:00.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:54:00.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:00.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:54:00.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:00.575 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:00.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:00.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:00.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:54:00.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:54:00.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:00.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:00.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:00.576 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:00.577 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:00.577 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:00.577 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:00.577 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:00.577 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:54:00.577 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:00.577 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:00.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:00.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:00.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:00.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:00.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:00.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:00.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:00.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:00.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:00.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:00.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:54:00.579 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:00.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=befbdd2c-0784-4f79-9a82-fa1876c3b073 2020-03-11T22:54:00.579 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 befbdd2c-0784-4f79-9a82-fa1876c3b073' 2020-03-11T22:54:00.580 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:00.580 INFO:tasks.workunit.client.0.smithi016.stdout:add osd0 befbdd2c-0784-4f79-9a82-fa1876c3b073 2020-03-11T22:54:00.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAIbGleVjo2IxAA9rQOxZ7O8HKOHFVUSswfBw== 2020-03-11T22:54:00.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAIbGleVjo2IxAA9rQOxZ7O8HKOHFVUSswfBw=="}' 2020-03-11T22:54:00.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new befbdd2c-0784-4f79-9a82-fa1876c3b073 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:54:00.966 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:54:00.975 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:54:00.976 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAIbGleVjo2IxAA9rQOxZ7O8HKOHFVUSswfBw== --osd-uuid befbdd2c-0784-4f79-9a82-fa1876c3b073 2020-03-11T22:54:01.001 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:00.998+0000 7f81dda54ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:01.002 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:01.000+0000 7f81dda54ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:01.005 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:01.002+0000 7f81dda54ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:01.005 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:01.002+0000 7f81dda54ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:01.005 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:01.003+0000 7f81dda54ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:01.006 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:01.003+0000 7f81dda54ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:01.055 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:01.052+0000 7f81dda54ec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:54:04.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:54:04.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:04.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:54:04.169 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:04.169 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd0 key to auth repository 2020-03-11T22:54:04.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:54:04.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:04.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:04.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:04.669 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:04.670 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.0 2020-03-11T22:54:04.691 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:04.688+0000 7f75a7391ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:04.693 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:04.690+0000 7f75a7391ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:04.695 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:04.692+0000 7f75a7391ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:05.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:54:05.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:05.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:54:05.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:05.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:05.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:05.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:05.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:05.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:54:05.035 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:54:05.213 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:05.209+0000 7f75a7391ec0 -1 Falling back to public interface 2020-03-11T22:54:05.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:06.010 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:06.007+0000 7f75a7391ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:54:06.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:06.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:06.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:06.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:06.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:54:06.404 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:54:06.782 INFO:tasks.workunit.client.0.smithi016.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85287,v1:127.0.0.1:6803/85287] [v2:127.0.0.1:6804/85287,v1:127.0.0.1:6805/85287] exists,up befbdd2c-0784-4f79-9a82-fa1876c3b073 2020-03-11T22:54:06.782 INFO:tasks.workunit.client.0.smithi016.stdout:add osd1 c524cd70-d7a3-4abf-8f64-6933d7e0d545 2020-03-11T22:54:06.782 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:06.782 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:06.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:06.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:06.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:54:06.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:06.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:54:06.783 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:06.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:54:06.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:06.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:06.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:06.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:06.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:54:06.784 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:54:06.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:06.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:06.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:06.785 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:06.785 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:06.785 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:06.786 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:06.786 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:06.786 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:54:06.786 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:06.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:06.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:06.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:06.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:06.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:06.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:06.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:06.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:06.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:06.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:06.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:06.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:54:06.788 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:06.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=c524cd70-d7a3-4abf-8f64-6933d7e0d545 2020-03-11T22:54:06.788 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 c524cd70-d7a3-4abf-8f64-6933d7e0d545' 2020-03-11T22:54:06.789 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:06.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAObGlejYU8LxAAzY8uYmFHR6VvLti7Gurdcw== 2020-03-11T22:54:06.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAObGlejYU8LxAAzY8uYmFHR6VvLti7Gurdcw=="}' 2020-03-11T22:54:06.795 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new c524cd70-d7a3-4abf-8f64-6933d7e0d545 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:54:07.348 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:54:07.358 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:54:07.359 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAObGlejYU8LxAAzY8uYmFHR6VvLti7Gurdcw== --osd-uuid c524cd70-d7a3-4abf-8f64-6933d7e0d545 2020-03-11T22:54:07.384 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:07.381+0000 7f68fadaeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:07.389 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:07.386+0000 7f68fadaeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:07.389 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:07.387+0000 7f68fadaeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:07.390 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:07.387+0000 7f68fadaeec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:07.390 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:07.387+0000 7f68fadaeec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:07.390 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:07.387+0000 7f68fadaeec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:07.463 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:07.461+0000 7f68fadaeec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:54:10.658 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:54:10.659 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:10.660 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd1 key to auth repository 2020-03-11T22:54:10.660 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:54:10.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:11.141 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.1 2020-03-11T22:54:11.141 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:54:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:11.142 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:11.162 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:11.160+0000 7f1b10703ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:11.165 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:11.162+0000 7f1b10703ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:11.166 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:11.164+0000 7f1b10703ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:11.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:54:11.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:11.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:54:11.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:11.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:11.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:11.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:11.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:11.506 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:54:11.506 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:54:11.681 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:11.678+0000 7f1b10703ec0 -1 Falling back to public interface 2020-03-11T22:54:11.863 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:12.422 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:12.419+0000 7f1b10703ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:54:12.864 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:12.865 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:12.865 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:12.865 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:54:12.866 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:54:12.866 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:13.250 INFO:tasks.workunit.client.0.smithi016.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/85642,v1:127.0.0.1:6811/85642] [v2:127.0.0.1:6812/85642,v1:127.0.0.1:6813/85642] exists,up c524cd70-d7a3-4abf-8f64-6933d7e0d545 2020-03-11T22:54:13.251 INFO:tasks.workunit.client.0.smithi016.stdout:add osd2 e53764df-64d2-462d-ae6d-617994de093c 2020-03-11T22:54:13.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:13.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:13.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:13.251 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:13.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:54:13.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:13.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:13.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:54:13.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:13.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:54:13.252 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:54:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:54:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:13.253 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:13.254 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:13.254 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:13.254 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:13.254 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:13.254 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:13.254 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:13.254 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:54:13.255 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:13.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:13.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:13.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:13.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:13.255 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:13.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:13.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:13.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:13.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:13.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:13.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:13.256 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:54:13.256 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:13.257 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e53764df-64d2-462d-ae6d-617994de093c 2020-03-11T22:54:13.257 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 e53764df-64d2-462d-ae6d-617994de093c' 2020-03-11T22:54:13.257 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:13.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAVbGleFKyWDxAA/OMqgGek6nOacRy5XrdfPw== 2020-03-11T22:54:13.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAVbGleFKyWDxAA/OMqgGek6nOacRy5XrdfPw=="}' 2020-03-11T22:54:13.264 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e53764df-64d2-462d-ae6d-617994de093c -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:54:13.847 INFO:tasks.workunit.client.0.smithi016.stdout:2 2020-03-11T22:54:13.857 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:54:13.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAVbGleFKyWDxAA/OMqgGek6nOacRy5XrdfPw== --osd-uuid e53764df-64d2-462d-ae6d-617994de093c 2020-03-11T22:54:13.883 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:13.880+0000 7f60e0703ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:13.904 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:13.901+0000 7f60e0703ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:13.905 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:13.902+0000 7f60e0703ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:13.905 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:13.903+0000 7f60e0703ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:13.905 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:13.903+0000 7f60e0703ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:13.906 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:13.903+0000 7f60e0703ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:13.970 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:13.968+0000 7f60e0703ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:54:17.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:54:17.448 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:17.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:54:17.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:17.449 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd2 key to auth repository 2020-03-11T22:54:17.932 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:54:17.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:17.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:17.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:17.933 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:17.933 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.2 2020-03-11T22:54:17.955 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:17.951+0000 7f01214e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:17.956 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:17.953+0000 7f01214e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:17.958 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:17.954+0000 7f01214e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:18.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:54:18.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:18.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:54:18.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:18.320 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:18.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:18.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:18.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:18.321 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:54:18.321 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:54:18.473 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:18.469+0000 7f01214e6ec0 -1 Falling back to public interface 2020-03-11T22:54:18.699 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:19.245 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:19.242+0000 7f01214e6ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:54:19.700 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:19.701 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:19.701 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:19.701 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:19.701 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:54:19.701 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:54:20.160 INFO:tasks.workunit.client.0.smithi016.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/85997,v1:127.0.0.1:6819/85997] [v2:127.0.0.1:6820/85997,v1:127.0.0.1:6821/85997] exists,up e53764df-64d2-462d-ae6d-617994de093c 2020-03-11T22:54:20.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:20.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:20.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:20.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:20.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:54:20.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:20.161 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:20.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:54:20.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:20.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:54:20.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:20.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:20.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:20.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:20.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:54:20.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:54:20.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:20.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:20.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:20.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:20.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:20.163 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:20.164 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:20.164 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:20.164 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:54:20.164 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:20.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:20.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:20.164 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:20.165 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:20.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:20.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:54:20.168 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:20.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d157bae0-b9f3-400a-8ce8-d2802f0ef3c5 2020-03-11T22:54:20.170 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 d157bae0-b9f3-400a-8ce8-d2802f0ef3c5' 2020-03-11T22:54:20.170 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:20.170 INFO:tasks.workunit.client.0.smithi016.stdout:add osd3 d157bae0-b9f3-400a-8ce8-d2802f0ef3c5 2020-03-11T22:54:20.174 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAcbGlekpdAChAAqUXl9Yfzjkn3TLD1Ku2GFA== 2020-03-11T22:54:20.175 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAcbGlekpdAChAAqUXl9Yfzjkn3TLD1Ku2GFA=="}' 2020-03-11T22:54:20.175 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d157bae0-b9f3-400a-8ce8-d2802f0ef3c5 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:54:20.690 INFO:tasks.workunit.client.0.smithi016.stdout:3 2020-03-11T22:54:20.700 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:54:20.701 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAcbGlekpdAChAAqUXl9Yfzjkn3TLD1Ku2GFA== --osd-uuid d157bae0-b9f3-400a-8ce8-d2802f0ef3c5 2020-03-11T22:54:20.726 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:20.722+0000 7fec22e67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:20.737 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:20.733+0000 7fec22e67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:20.738 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:20.735+0000 7fec22e67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:20.739 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:20.735+0000 7fec22e67ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:20.739 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:20.735+0000 7fec22e67ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:20.739 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:20.735+0000 7fec22e67ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:20.837 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:20.833+0000 7fec22e67ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:54:23.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:54:23.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:23.957 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:54:23.958 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:23.958 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd3 key to auth repository 2020-03-11T22:54:24.448 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.3 2020-03-11T22:54:24.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:54:24.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:24.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:24.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:24.449 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:24.471 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:24.467+0000 7f75a4fa6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:24.473 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:24.469+0000 7f75a4fa6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:24.474 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:24.471+0000 7f75a4fa6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:24.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:54:24.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:24.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:54:24.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:24.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:24.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:24.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:24.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:24.850 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:54:24.850 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:54:24.988 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:24.984+0000 7f75a4fa6ec0 -1 Falling back to public interface 2020-03-11T22:54:25.209 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:25.729 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:25.726+0000 7f75a4fa6ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:54:26.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:26.210 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:26.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:26.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:26.211 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:54:26.211 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:54:26.605 INFO:tasks.workunit.client.0.smithi016.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/86352,v1:127.0.0.1:6827/86352] [v2:127.0.0.1:6828/86352,v1:127.0.0.1:6829/86352] exists,up d157bae0-b9f3-400a-8ce8-d2802f0ef3c5 2020-03-11T22:54:26.605 INFO:tasks.workunit.client.0.smithi016.stdout:add osd4 3381c23a-9b5d-444c-a55a-1d7c46b6bd44 2020-03-11T22:54:26.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:26.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:26.605 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:26.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:26.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:54:26.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:26.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:26.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:54:26.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:26.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:54:26.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:26.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:26.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:26.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:26.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:54:26.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:54:26.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:26.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:26.608 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:26.608 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:26.609 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:26.609 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:26.609 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:26.609 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:26.609 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:54:26.609 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:26.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:26.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:26.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:26.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:26.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:26.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:26.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:26.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:26.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:26.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:26.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:26.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:54:26.611 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:26.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3381c23a-9b5d-444c-a55a-1d7c46b6bd44 2020-03-11T22:54:26.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 3381c23a-9b5d-444c-a55a-1d7c46b6bd44' 2020-03-11T22:54:26.612 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:26.618 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAibGleML62JBAAc7FS3Ox+7s3T8XkjQ9L2CQ== 2020-03-11T22:54:26.619 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAibGleML62JBAAc7FS3Ox+7s3T8XkjQ9L2CQ=="}' 2020-03-11T22:54:26.619 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3381c23a-9b5d-444c-a55a-1d7c46b6bd44 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:54:27.166 INFO:tasks.workunit.client.0.smithi016.stdout:4 2020-03-11T22:54:27.176 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:54:27.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAibGleML62JBAAc7FS3Ox+7s3T8XkjQ9L2CQ== --osd-uuid 3381c23a-9b5d-444c-a55a-1d7c46b6bd44 2020-03-11T22:54:27.201 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:27.197+0000 7f8a6ae72ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:27.202 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:27.199+0000 7f8a6ae72ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:27.205 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:27.201+0000 7f8a6ae72ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:27.205 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:27.201+0000 7f8a6ae72ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:27.205 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:27.201+0000 7f8a6ae72ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:27.205 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:27.201+0000 7f8a6ae72ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:27.287 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:27.283+0000 7f8a6ae72ec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:54:30.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:54:30.674 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:30.675 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd4 key to auth repository 2020-03-11T22:54:30.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:54:30.676 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:31.159 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.4 2020-03-11T22:54:31.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:54:31.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:31.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:31.181 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:31.177+0000 7f6b452c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:31.183 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:31.180+0000 7f6b452c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:31.185 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:31.181+0000 7f6b452c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:31.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:54:31.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:31.523 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:54:31.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:31.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:31.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:31.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:31.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:31.524 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:31.525 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:54:31.700 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:31.697+0000 7f6b452c1ec0 -1 Falling back to public interface 2020-03-11T22:54:31.883 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:32.488 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:32.485+0000 7f6b452c1ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:54:32.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:32.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:32.885 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:32.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:32.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:32.886 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:54:33.267 INFO:tasks.workunit.client.0.smithi016.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/86707,v1:127.0.0.1:6835/86707] [v2:127.0.0.1:6836/86707,v1:127.0.0.1:6837/86707] exists,up 3381c23a-9b5d-444c-a55a-1d7c46b6bd44 2020-03-11T22:54:33.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:33.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:33.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:33.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:33.267 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:54:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:54:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:54:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:33.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:33.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:33.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:54:33.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:54:33.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:33.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:33.269 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:33.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:33.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:33.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:33.270 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:33.270 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:33.270 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76802 2020-03-11T22:54:33.270 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' 2020-03-11T22:54:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:33.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:33.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:33.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:33.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:33.272 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:54:33.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:33.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3501b449-bdfd-4f31-9b95-09ecdf869bda 2020-03-11T22:54:33.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 3501b449-bdfd-4f31-9b95-09ecdf869bda' 2020-03-11T22:54:33.278 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:33.279 INFO:tasks.workunit.client.0.smithi016.stdout:add osd5 3501b449-bdfd-4f31-9b95-09ecdf869bda 2020-03-11T22:54:33.289 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQApbGleWDsUERAA5XG1sSURgsLziFBt3QU6+w== 2020-03-11T22:54:33.289 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQApbGleWDsUERAA5XG1sSURgsLziFBt3QU6+w=="}' 2020-03-11T22:54:33.289 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3501b449-bdfd-4f31-9b95-09ecdf869bda -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:33.759 INFO:tasks.workunit.client.0.smithi016.stdout:5 2020-03-11T22:54:33.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:33.770 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQApbGleWDsUERAA5XG1sSURgsLziFBt3QU6+w== --osd-uuid 3501b449-bdfd-4f31-9b95-09ecdf869bda 2020-03-11T22:54:33.794 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:33.790+0000 7f2aa2676ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:33.796 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:33.792+0000 7f2aa2676ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:33.798 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:33.794+0000 7f2aa2676ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:33.798 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:33.795+0000 7f2aa2676ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:33.798 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:33.795+0000 7f2aa2676ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:33.799 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:33.795+0000 7f2aa2676ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:33.937 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:33.933+0000 7f2aa2676ec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:54:37.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:54:37.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:37.411 INFO:tasks.workunit.client.0.smithi016.stdout:adding osd5 key to auth repository 2020-03-11T22:54:37.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:54:37.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:37.894 INFO:tasks.workunit.client.0.smithi016.stdout:start osd.5 2020-03-11T22:54:37.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:54:37.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=4905196d-3e76-4e16-ace8-ee1dceef9d45 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:37.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:37.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:37.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:37.917 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:37.913+0000 7fb7a2b69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:37.919 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:37.915+0000 7fb7a2b69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:37.920 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:37.916+0000 7fb7a2b69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:38.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:54:38.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:38.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:54:38.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:38.259 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:38.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:38.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:38.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:38.260 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:38.260 INFO:tasks.workunit.client.0.smithi016.stdout:0 2020-03-11T22:54:38.435 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:38.431+0000 7fb7a2b69ec0 -1 Falling back to public interface 2020-03-11T22:54:38.620 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:39.243 INFO:tasks.workunit.client.0.smithi016.stderr:2020-03-11T22:54:39.239+0000 7fb7a2b69ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:54:39.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:39.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:39.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:39.622 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:39.623 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:39.623 INFO:tasks.workunit.client.0.smithi016.stdout:1 2020-03-11T22:54:40.055 INFO:tasks.workunit.client.0.smithi016.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/87062,v1:127.0.0.1:6843/87062] [v2:127.0.0.1:6844/87062,v1:127.0.0.1:6845/87062] exists,up 3501b449-bdfd-4f31-9b95-09ecdf869bda 2020-03-11T22:54:40.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:40.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:40.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:40.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:54:40.056 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:54:40.695 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'test' already exists 2020-03-11T22:54:40.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:54:41.704 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:54:41.705 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:41.705 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:54:42.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:42.058 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:54:42.159 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:42.160 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:54:42.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:42.461 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:54:42.695 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:42.696 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:54:42.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:42.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:54:42.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:42.817 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:54:42.867 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:42.868 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:54:42.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:42.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:54:43.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:54:43.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:54:43.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.263 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:54:43.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.318 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:54:43.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.388 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:54:43.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.443 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:54:43.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:54:43.567 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.568 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:54:43.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.655 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:54:43.847 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:54:43.901 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.902 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:54:43.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:43.952 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:54:44.001 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.002 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:54:44.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:54:44.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.177 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:54:44.340 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.340 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:54:44.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.457 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:54:44.518 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.519 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:54:44.576 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.577 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:54:44.652 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.652 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:54:44.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.702 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:54:44.751 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.752 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:54:44.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:44.895 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:54:45.017 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.017 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:54:45.069 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.069 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:54:45.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.192 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:54:45.243 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.244 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:54:45.293 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.294 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:54:45.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.344 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:54:45.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.476 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:54:45.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.593 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:54:45.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.727 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:54:45.785 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:54:45.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.836 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:54:45.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.886 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:54:45.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:45.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:54:46.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.060 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:54:46.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:54:46.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.353 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:54:46.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.402 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:54:46.452 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.453 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:54:46.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.502 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:54:46.577 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.578 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:54:46.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:54:46.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.826 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:54:46.887 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.888 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:54:46.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:54:46.986 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:46.986 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:54:47.036 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.036 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:54:47.120 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.120 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:54:47.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.213 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:54:47.313 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.313 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:54:47.447 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.447 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:54:47.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.503 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:54:47.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.553 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:54:47.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.604 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:54:47.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.678 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:54:47.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:54:47.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.873 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:54:47.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:47.990 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:54:48.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.045 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:54:48.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.095 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:54:48.228 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.229 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:54:48.278 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.279 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:54:48.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:54:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:54:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:54:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:54:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:54:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:54:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:54:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:54:49.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:54:49.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.168 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:54:49.220 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.221 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:54:49.360 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.361 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:54:49.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:54:49.521 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.521 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:54:49.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.720 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:54:49.878 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.878 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:54:49.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.929 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:54:49.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.979 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:54:50.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:54:50.080 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.080 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:54:50.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.188 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:54:50.304 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.304 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:54:50.438 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.438 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:54:50.487 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.488 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:54:50.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.538 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:54:50.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.588 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:54:50.638 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.638 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:54:50.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:54:50.871 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.872 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:54:51.005 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:54:51.054 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.055 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:54:51.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.105 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:54:51.189 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.189 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:54:51.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.314 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:54:51.431 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.432 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:54:51.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:54:51.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.615 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:54:51.671 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.672 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:54:51.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.722 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:54:51.771 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:54:51.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:54:52.007 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.008 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:54:52.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:54:52.197 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.197 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:54:52.247 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.247 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:54:52.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:54:52.441 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:54:52.533 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:54:52.692 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.692 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:54:52.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.755 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:54:52.805 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.806 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:54:52.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:54:52.905 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:54:52.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.989 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:54:53.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.076 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:54:53.236 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.236 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:54:53.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.297 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:54:53.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.347 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:54:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:54:53.455 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.456 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:54:53.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.534 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:54:53.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.611 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:54:53.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:54:53.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.809 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:54:53.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.856 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:54:53.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.906 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:54:53.956 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:53.957 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:54:54.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.006 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:54:54.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:54:54.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:54:54.389 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.390 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:54:54.439 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.440 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:54:54.489 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.490 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:54:54.617 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.617 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:54:54.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.673 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:54:54.762 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:54.763 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:54:54.922 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:54.922 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:54:55.265 INFO:tasks.workunit.client.0.smithi016.stderr:dumped pgs 2020-03-11T22:54:55.274 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:54:55.844 INFO:tasks.workunit.client.0.smithi016.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:54:55.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:54:55.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:54:56.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:54:56.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:54:56.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:54:56.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:54:56.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:54:56.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:54:56.856 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:54:56.856 INFO:tasks.workunit.client.0.smithi016.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:54:56.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:54:56.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:54:56.856 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:54:56.961 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:54:56.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:54:56.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:54:56.962 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:56.962 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi016.stderr:1 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi016.stderr:2 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi016.stderr:3 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi016.stderr:4 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi016.stderr:5' 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:57.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:54:57.464 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:54:57.465 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:57.465 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:57.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:54:57.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2020-03-11T22:54:57.594 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:57.595 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:57.724 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509450 2020-03-11T22:54:57.724 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450' 2020-03-11T22:54:57.724 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:57.724 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:57.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345929 2020-03-11T22:54:57.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929' 2020-03-11T22:54:57.855 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:57.855 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:57.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182407 2020-03-11T22:54:57.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929 4-107374182407' 2020-03-11T22:54:57.983 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:57.983 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:54:58.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=124554051590 2020-03-11T22:54:58.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450 3-85899345929 4-107374182407 5-124554051590' 2020-03-11T22:54:58.113 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:58.113 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:54:58.113 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:58.114 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:58.114 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:54:58.114 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:58.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:54:58.115 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:54:58.116 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:58.116 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:54:58.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:54:58.482 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:58.484 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:54:58.484 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:58.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:58.484 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:54:58.484 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:58.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:54:58.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:54:58.485 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:58.485 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:54:58.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672971 2020-03-11T22:54:58.848 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:58.849 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509450 2020-03-11T22:54:58.849 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:58.849 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:58.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509450 2020-03-11T22:54:58.850 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:58.851 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509450 2020-03-11T22:54:58.851 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-03-11T22:54:58.852 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:58.852 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.2 seq 64424509450 2020-03-11T22:54:59.236 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509450 2020-03-11T22:54:59.236 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:59.237 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345929 2020-03-11T22:54:59.237 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:59.237 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:59.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345929 2020-03-11T22:54:59.238 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:59.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345929 2020-03-11T22:54:59.239 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2020-03-11T22:54:59.239 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:59.239 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.3 seq 85899345929 2020-03-11T22:54:59.658 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345929 -lt 85899345929 2020-03-11T22:54:59.658 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:59.658 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182407 2020-03-11T22:54:59.659 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:59.659 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:59.659 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182407 2020-03-11T22:54:59.660 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:59.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182407 2020-03-11T22:54:59.661 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182407' 2020-03-11T22:54:59.661 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:59.661 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.4 seq 107374182407 2020-03-11T22:55:00.026 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182407 -lt 107374182407 2020-03-11T22:55:00.027 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:55:00.027 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-124554051590 2020-03-11T22:55:00.027 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:55:00.028 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:55:00.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-124554051590 2020-03-11T22:55:00.028 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:55:00.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=124554051590 2020-03-11T22:55:00.029 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 124554051590' 2020-03-11T22:55:00.029 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:55:00.030 INFO:tasks.workunit.client.0.smithi016.stdout:waiting osd.5 seq 124554051590 2020-03-11T22:55:00.460 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 124554051590 -lt 124554051590 2020-03-11T22:55:00.460 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:55:00.460 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:00.461 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:00.935 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:55:00.936 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:00.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:00.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:00.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:00.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:00.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:00.936 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:01.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:01.768 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:55:01.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:55:01.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:55:01.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:01.769 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:01.869 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:01.870 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:01.870 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:01.870 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:01.870 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:01.870 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:01.870 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:01.871 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:02.218 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:02.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:02.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:02.219 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:02.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:02.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:02.688 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:02.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:02.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:02.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:02.689 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:02.689 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:02.689 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:55:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:55:03.162 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 13 )) 2020-03-11T22:55:03.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:03.163 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:55:03.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:03.366 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:03.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:03.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:03.367 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:03.368 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:03.368 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:03.368 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:03.712 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:03.712 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:03.713 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:03.713 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:04.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:04.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:04.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:04.194 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:04.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:04.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:04.195 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:04.195 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:04.195 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:05.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23325 2020-03-11T22:55:05.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23325 '!=' null 2020-03-11T22:55:05.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:05.085 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:05.086 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:05.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:05.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:05.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:05.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:05.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:05.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:05.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:05.187 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:05.559 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:05.559 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:05.559 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:05.560 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:06.034 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:06.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:06.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:06.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:06.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:06.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:06.035 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:06.035 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:06.036 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:06.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19112 2020-03-11T22:55:06.530 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19112 '!=' null 2020-03-11T22:55:06.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:06.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:06.531 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:06.631 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:06.631 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:06.985 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:06.985 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:06.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:06.986 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:07.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:07.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:07.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:07.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:07.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:07.470 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:07.471 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:07.471 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:07.471 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:07.969 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19112 2020-03-11T22:55:07.970 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19112 '!=' null 2020-03-11T22:55:07.970 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:07.970 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:07.970 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:08.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:08.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:08.071 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:08.071 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:08.071 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:08.072 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:08.072 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:08.072 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:08.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:08.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:08.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:08.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:08.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:08.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:08.922 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:08.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:08.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:08.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:08.923 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:08.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:08.923 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:09.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31376 2020-03-11T22:55:09.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31376 '!=' null 2020-03-11T22:55:09.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:09.409 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:09.410 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:09.510 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:09.510 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:09.510 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:09.510 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:09.510 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:09.510 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:09.511 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:09.511 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:09.860 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:09.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:09.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:09.861 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:10.382 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:10.382 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:10.382 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:10.383 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:10.383 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:10.383 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:10.383 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:10.383 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:10.383 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:10.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29899 2020-03-11T22:55:10.853 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29899 '!=' null 2020-03-11T22:55:10.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:10.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:10.854 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:10.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:10.955 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:10.955 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:10.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:10.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:10.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:10.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:10.956 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:11.311 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:11.311 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:11.312 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:11.312 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:11.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:11.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:11.786 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:11.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:11.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:11.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:11.787 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:11.787 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:11.787 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:12.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29899 2020-03-11T22:55:12.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29899 '!=' null 2020-03-11T22:55:12.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:12.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:12.341 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:12.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:12.442 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:12.442 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:12.443 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:12.443 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:12.443 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:12.443 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:12.443 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:12.796 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:12.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:12.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:12.796 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:13.275 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:13.275 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:13.275 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:13.275 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:13.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:13.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:13.276 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:13.276 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:13.276 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:13.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29899 2020-03-11T22:55:13.772 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29899 '!=' null 2020-03-11T22:55:13.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:13.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:13.773 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:13.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:13.874 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:13.874 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:13.874 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:13.874 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:13.875 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:13.875 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:13.875 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:14.235 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:14.235 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:14.236 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:14.236 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:14.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:14.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:14.718 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:14.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:14.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:14.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:14.719 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:14.720 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:14.720 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:15.186 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45423 2020-03-11T22:55:15.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45423 '!=' null 2020-03-11T22:55:15.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:15.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:15.187 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:15.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:15.287 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:15.288 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:15.288 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:15.289 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:15.289 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:15.289 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:15.289 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:16.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:16.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:16.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:16.138 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:16.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:16.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:16.139 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:16.139 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:16.140 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:16.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29899 2020-03-11T22:55:16.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29899 '!=' null 2020-03-11T22:55:16.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:16.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:16.610 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:16.710 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:16.711 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:16.712 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:16.712 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:16.712 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:16.712 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:16.712 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:16.713 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:17.063 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:17.063 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:17.064 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:17.064 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:17.537 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:17.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:17.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:17.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:17.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:17.606 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:17.607 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:17.607 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:17.607 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:18.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29899 2020-03-11T22:55:18.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29899 '!=' null 2020-03-11T22:55:18.071 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:18.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:18.072 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:18.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:18.172 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:18.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:18.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:18.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:18.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:18.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:18.173 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:18.526 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:18.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:18.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:18.527 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:19.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:19.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:19.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:19.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:19.003 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:19.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:19.004 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:19.004 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:19.004 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:19.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45998 2020-03-11T22:55:19.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45998 '!=' null 2020-03-11T22:55:19.484 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:19.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:19.485 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:19.585 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:19.586 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:19.586 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:19.586 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:19.586 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:19.586 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:19.586 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:19.586 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:19.939 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:19.939 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:19.939 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:19.940 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:20.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:20.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:20.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:20.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:20.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:20.437 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:20.438 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:20.438 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:20.438 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:20.950 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31623 2020-03-11T22:55:20.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31623 '!=' null 2020-03-11T22:55:20.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:20.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:20.951 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:21.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:21.052 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:21.052 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:21.052 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:21.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:21.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:21.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:21.053 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:21.411 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:21.411 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:21.412 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:21.412 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:21.881 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:21.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:21.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:21.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:21.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:21.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:21.882 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:21.882 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:21.883 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:22.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31624 2020-03-11T22:55:22.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31624 '!=' null 2020-03-11T22:55:22.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:22.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:22.404 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:22.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:22.505 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:22.505 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:22.505 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:22.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:22.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:22.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:22.506 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:22.858 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:22.858 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:22.858 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:22.859 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:23.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:23.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:23.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:23.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:23.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:23.333 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:23.334 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:23.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:23.334 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:23.851 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31624 2020-03-11T22:55:23.851 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31624 '!=' null 2020-03-11T22:55:23.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:23.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:23.852 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:23.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:23.948 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:23.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:23.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:23.949 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:23.950 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:23.950 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:23.950 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:24.302 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:24.302 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:24.302 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:24.302 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:24.778 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:24.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:24.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:24.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:24.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:24.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:24.779 INFO:tasks.workunit.client.0.smithi016.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:24.779 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:24.780 INFO:tasks.workunit.client.0.smithi016.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:24.950 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-11T22:55:24.950 DEBUG:teuthology.task.console_log:Killing console logger for smithi016 2020-03-11T22:55:24.951 DEBUG:teuthology.task.console_log:Killing console logger for smithi016 2020-03-11T22:55:24.951 DEBUG:teuthology.exit:Finished running handlers